7 ms·

A non-marketing example: We were onboarding a large new client to our SAAS product. This process involved creating accounts for all of their employees (tens of

A non-marketing example: We were onboarding a large new client to our SAAS product. This process involved creating accounts for all of their employees (tens of thousands) and sending emails with an activation link. (Where they'd be able to set up their password.) Our system sends these emails in batches, and as soon as the first batch went out we got an alert from our monitoring system that our bounce rate was surging - high enough to risk a sending pause from Amazon SES. We stopped sending and investigated the issue, and it turned out that the email list we were given was a mess - it included all current employees, but also a huge number of former ones. Just under 1/10th of the emails in our first batch were invalid. We asked the client to give us a better list, but due to internal issues they couldn't get that to us any time soon. Meanwhile they were breathing down our necks to get these emails out ASAP, and they were a large enough client that our management wanted to keep them happy, so we tried out one of these email validation services. Unfortunately, it didn't work. It turns out that this technique doesn't work for all mail servers. It was reporting every email as valid, even ones we knew were invalid since they'd already hard bounced. (Edit: thinking back - this was several years ago - I think it wasn't saying that they were valid emails, just that it couldn't tell whether they were valid or not - the service was able to detect that the server wasn't rejecting non-existent addresses.) We ended up unpausing the emails and just hoping for the best. Ended up with something like an 8% bounce rate that eventually fell off our record as our normal sending patterns resumed. Amazon's guidelines say they might cut you off when you hit 10%, so we cut it pretty close.

donmcronald 5y ago
It seems like the most practical solution to that should be calling AWS, explaining it for 5 min, and getting an exception. Is that kind of reasonable solution no longer possible with the cloud providers being so huge?
- toomanybeersies 5y ago
  It's possible, you just need to pay the extra 10% for premium support.
kapp_in_life 5y ago
I'd hope you'd at least be able to explain the situation to your account manager and get an exception(maybe for that single companies domain?), but I've never used AWS so I wouldn't know if thats possible.
- polynomial 5y ago
  account manager? you fancy.
  - isbvhodnvemrwvn 5y ago
    TAMs you get with the higher support plans can be of some help in situations like these.
femto113 5y ago
If they're checking using SMTP's VRFY command then it's actually considered a best practice for the server to always reply with a 252 "cannot verify" since otherwise it can be used to fish for valid addresses.
- sildur 5y ago
  But... I could also fish for valid addresses by trying to send an email and waiting for a bounce...
  - e12e 5y ago
    Yes, and you could be eg: graylisted. Or server could accept all an silently drop. Or rate limit. It's sad that VERIFY is basically dead due to spam. In the olden days, you might have been able to use finger - but it's also dead for (among other) similar reasons. The spam cat and mouse game leads to quite silly situations for benign actors. For some mx's you might be able to designate some ips as trusted, and do real verify for those.
bkuehl 5y ago
I can't even imagine wanting to handle managing accounts and credentials for that many users at an enterprise! At that point SSO integration is well worth the money. How did you handle removing access when a user was no longer employed at the company?
- moooo99 5y ago
  Not OP, but also building a similar user system. I can totally understand the motivation to not use the internal SSO. With most companies I know, as soon as you actually connect to their private datasources, you have to do some extra steps to prove how you're securing your platform. This makes sense from the companies perspective, but also introduces a huge technical and organizational overhead for the startup which might be better spend elsewhere if your product does not absolutely rely on SSO
killingtime74 5y ago
I’ve never worked with emails, could you not send these first emails yourself and not use SES
- EnderWT 5y ago
  Have you looked into what SES or other email services provide? Sending emails is easy, while actually getting them delivered is harder. You have to make sure you're not getting flagged as spam, can handle bouncebacks, etc. Here's one discussion: https://stackoverflow.com/questions/371/how-do-you-make-sure-email-you-send-programmatically-is-not-automatically-marked https://stackoverflow.com/questions/371/how-do-you-make-sure...
  - indigo945 5y ago
    Not getting flagged as spam isn't actually that hard, though. Besides, if you're using SES or some other hosted SMTP service, you still have to set up SPF for your domain, so you haven't even really gained much comfort. The only really useful thing is to gain an ip address with a high reputation, but you can generally get those at any reputable hosting provider as well. Just don't try sending emails from your residential internet connection.
  - toomanybeersies 5y ago
    In GP's case, they probably could'e arranged with the client's sysadmins to whitelist the IP they were sending emails from.
  - killingtime74 5y ago
    Ah I understand that part but if it’s going to a big client who knows the emails are coming is it besides the point? I understand day to day
ganafagol 5y ago
This is clearly trying to solve a non-technical problem with technical means. The root problem is that AWS cancels you with too high bounce rate. The obvious solution is to talk to some AWS representative to at least temporarily not cancel you after explaining the situation. If AWS does not let you talk to them, then that's where the problem lies, not in some not cleaned up email list. It's terrible to spend a lot of effort on this kind of tech just because some business partner has shitty customer support.
- aembleton 5y ago
  But that would affect all customers of Amazon SES because the IP addresses would score low by other email servers.
  - kozziollek 5y ago
    Assuming that the customer's servers were hosted by some SaaS like Office, right? If customer was hosting e-mail servers themselves they wouldn't notify any other mail servers?
- golergka 5y ago
  > This is clearly trying to solve a non-technical problem with technical means. Isn't that the whole point of what most of us are doing?
  - ganafagol 5y ago
    Is it? In my dayjob I'm solving technical problems with technical means. World hunger is not a technical problem. You won't solve it with technical means. If you think you can, you have already lost the fight. Climate change is not a technical problem. You won't solve it with technical means. If you think you can, you have already lost the fight. And so on, and so forth. Technical means can help solving certain components needed for the overall solution. These are then technical (sub)problems though. For example, how to store more energy in a battery, or how to grow certain crops with less water. But the overall problems are social in nature. People need to understand that world hunger is a distribution problem. That one is easier to solve with certain (technical) tools available, but that won't be enough. People need to understand that we can't use more natural resources than get replenished. Not a technical problem. If only the tools get better, people will find new ways to be wasteful. Etc etc.
justinator 5y ago
Why did all the employees have email addresses from different services, rather than a single @ourcompany.com address pool that they controlled?
friendzis 5y ago
At that point would it not be easier to just spin up a VM in customer's infra and send emails directly to their exchange server?
toomanybeersies 5y ago
Wouldn't the better solution have been to support SSO?
kortilla 5y ago
This is a reason you need an escape hatch from SES. In the past when I worked on a system that needed to notify via email we always had a way to change delivery process for certain emails, domains, etc for exactly this reason. This is one of those cases where we would “deliver directly” (i.e. send directly to their mail provider).
geoduck14 5y ago
Ok, so first pass gives me a TERRIBLE idea, that would "get the job done". I'm sure you thought of this and dismissed it: Keep your "overall bounce rate" low, by ALSO sending out extra emails to confirmed email addresses. Like, for every "confirmation" email, also send a "thanks for joining us" email to someone that already confirmed their email.

donmcronald 5y ago

It seems like the most practical solution to that should be calling AWS, explaining it for 5 min, and getting an exception. Is that kind of reasonable solution no longer possible with the cloud providers being so huge?

toomanybeersies 5y ago
It's possible, you just need to pay the extra 10% for premium support.

kapp_in_life 5y ago

I'd hope you'd at least be able to explain the situation to your account manager and get an exception(maybe for that single companies domain?), but I've never used AWS so I wouldn't know if thats possible.

polynomial 5y ago
account manager? you fancy.
- isbvhodnvemrwvn 5y ago
  TAMs you get with the higher support plans can be of some help in situations like these.

femto113 5y ago

If they're checking using SMTP's VRFY command then it's actually considered a best practice for the server to always reply with a 252 "cannot verify" since otherwise it can be used to fish for valid addresses.

sildur 5y ago
But... I could also fish for valid addresses by trying to send an email and waiting for a bounce...
- e12e 5y ago
  Yes, and you could be eg: graylisted. Or server could accept all an silently drop. Or rate limit. It's sad that VERIFY is basically dead due to spam. In the olden days, you might have been able to use finger - but it's also dead for (among other) similar reasons. The spam cat and mouse game leads to quite silly situations for benign actors. For some mx's you might be able to designate some ips as trusted, and do real verify for those.

bkuehl 5y ago

I can't even imagine wanting to handle managing accounts and credentials for that many users at an enterprise! At that point SSO integration is well worth the money. How did you handle removing access when a user was no longer employed at the company?

moooo99 5y ago
Not OP, but also building a similar user system. I can totally understand the motivation to not use the internal SSO. With most companies I know, as soon as you actually connect to their private datasources, you have to do some extra steps to prove how you're securing your platform. This makes sense from the companies perspective, but also introduces a huge technical and organizational overhead for the startup which might be better spend elsewhere if your product does not absolutely rely on SSO

killingtime74 5y ago

I’ve never worked with emails, could you not send these first emails yourself and not use SES

EnderWT 5y ago
Have you looked into what SES or other email services provide? Sending emails is easy, while actually getting them delivered is harder. You have to make sure you're not getting flagged as spam, can handle bouncebacks, etc. Here's one discussion: https://stackoverflow.com/questions/371/how-do-you-make-sure-email-you-send-programmatically-is-not-automatically-marked https://stackoverflow.com/questions/371/how-do-you-make-sure...
- indigo945 5y ago
  Not getting flagged as spam isn't actually that hard, though. Besides, if you're using SES or some other hosted SMTP service, you still have to set up SPF for your domain, so you haven't even really gained much comfort. The only really useful thing is to gain an ip address with a high reputation, but you can generally get those at any reputable hosting provider as well. Just don't try sending emails from your residential internet connection.
- toomanybeersies 5y ago
  In GP's case, they probably could'e arranged with the client's sysadmins to whitelist the IP they were sending emails from.
- killingtime74 5y ago
  Ah I understand that part but if it’s going to a big client who knows the emails are coming is it besides the point? I understand day to day

ganafagol 5y ago

This is clearly trying to solve a non-technical problem with technical means. The root problem is that AWS cancels you with too high bounce rate. The obvious solution is to talk to some AWS representative to at least temporarily not cancel you after explaining the situation. If AWS does not let you talk to them, then that's where the problem lies, not in some not cleaned up email list. It's terrible to spend a lot of effort on this kind of tech just because some business partner has shitty customer support.

aembleton 5y ago
But that would affect all customers of Amazon SES because the IP addresses would score low by other email servers.
- kozziollek 5y ago
  Assuming that the customer's servers were hosted by some SaaS like Office, right? If customer was hosting e-mail servers themselves they wouldn't notify any other mail servers?
golergka 5y ago
> This is clearly trying to solve a non-technical problem with technical means. Isn't that the whole point of what most of us are doing?
- ganafagol 5y ago
  Is it? In my dayjob I'm solving technical problems with technical means. World hunger is not a technical problem. You won't solve it with technical means. If you think you can, you have already lost the fight. Climate change is not a technical problem. You won't solve it with technical means. If you think you can, you have already lost the fight. And so on, and so forth. Technical means can help solving certain components needed for the overall solution. These are then technical (sub)problems though. For example, how to store more energy in a battery, or how to grow certain crops with less water. But the overall problems are social in nature. People need to understand that world hunger is a distribution problem. That one is easier to solve with certain (technical) tools available, but that won't be enough. People need to understand that we can't use more natural resources than get replenished. Not a technical problem. If only the tools get better, people will find new ways to be wasteful. Etc etc.

justinator 5y ago

Why did all the employees have email addresses from different services, rather than a single @ourcompany.com address pool that they controlled?

friendzis 5y ago

At that point would it not be easier to just spin up a VM in customer's infra and send emails directly to their exchange server?

toomanybeersies 5y ago

Wouldn't the better solution have been to support SSO?

kortilla 5y ago

This is a reason you need an escape hatch from SES. In the past when I worked on a system that needed to notify via email we always had a way to change delivery process for certain emails, domains, etc for exactly this reason. This is one of those cases where we would “deliver directly” (i.e. send directly to their mail provider).

geoduck14 5y ago

Ok, so first pass gives me a TERRIBLE idea, that would "get the job done". I'm sure you thought of this and dismissed it: Keep your "overall bounce rate" low, by ALSO sending out extra emails to confirmed email addresses. Like, for every "confirmation" email, also send a "thanks for joining us" email to someone that already confirmed their email.