4 ms·

I think it's clearly better to have the email eventually sent than the case where a completely synchronous service is down resulting in no email being sent

iudqnolq 3y ago
Under certain circumstances you might want to put backpressure on the email emitter. Under certain circumstances you might want to hard error. If I'm trying to reset my password and your UI says to check my email but I don't receive an email I'm gonna be annoyed. My first guess will be some kind of intermittent delivery issue, so I'll probably send a new request to your UI. If this is important to me then after an hour or so I might open a support ticket. I'll have wasted a bunch of time checking to see if I've finally got the email. I'd much rather have gotten an error and the retried when your status page indicates the issue is resolved.
- lutzh 3y ago
  Thanks for your feedback! Delayed e-mail can have of course have many other reasons than the service supposed to send it being down. But let's go with your example. Let's say you have a LoginService, and a MessagingService, and if you request a password reset, the LoginService sends a ResetRequested event. Now if the MessagingService is down, you'd still get told to check your inbox, although the e-mail hasn't even been sent. That's a bad user experience. Agreed. Your suggested solution is to make the communication between LoginService and MessagingService not event driven. So if the MessagingService is down, you can "hard error". But how much better is the experience then, really? As a user, you request the reset, and after a few seconds (you're already annoyed and checking if your internet connection is down), you get an error message that you can't reset your password right now. I'd argue that option is equally unacceptable. What you need to do, is to make the MessagingService better. In an event driven system, each participating service makes a promise. The MessagingService must give service level guarantees, it must be built and scaled to be able to process all the events it's listening to. If it has an error, not you should get the error message - you are the last person able to do something about it - but someone who fixes it. If you want a better user experience, don't build unreliable services and then apologize for the inconvenience. Built better (more reliable, more scalable) software. In the end, that's what event-driven architecture is about.
  - iudqnolq 3y ago
    > Delayed e-mail can have of course have many other reasons than the service supposed to send it being down. Of course. But I was presuming that your email service makes an API call to an email service provider rather than speaking SMTP itself. That API call has three possible cases: - Ok: You're done, your ESP is responsible for SMTP retries. Maybe there's a webhook to re-enqueue something eventually after failures. - Transient error: You retry. But you probably don't want to wait that long and there's probably no point in interleaving attempts to send other emails through the same API. So you may as well "block" (as in yield to the async scheduler) while waiting to retry. - Persistent error: Here's when I suggest hard failing up to the user. > If it has an error, not you [the end user] should get the error message - you are the last person able to do something about it - but someone who fixes it. In the post you replied to I walked through in detail how my experience would be noticably better if I got an error message. > If you want a better user experience, don't build unreliable services and then apologize for the inconvenience. Built better (more reliable, more scalable) software. I agree you should attempt to eliminate all failability. I disagree that if you think your software is perfect you don't need to write code that provides the least unpleasant user experience possible if an error happens. Email is an exceptional example: it is theoretically impossible to write a service that infallibly delivers email to {gmail,yahoo,outlook} within a short period of time.
- ericbarrett 3y ago
  MessagingService includes its queue depth, or age, to LoginService in the enqueue response. If over $threshold, LoginService tells UI to add a parenthetical "(our email is backed up but you'll get it soon)" notice. Add alerting around high queue depth and I think you've buttoned this up.
  - iudqnolq 3y ago
    Agree that's another way to solve it. Treating password reset emails as transient, erroring, and not enqueuing would also work and might be simpler.
    - ericbarrett 3y ago
      For sure, guess it depends on the volume of the service and balancing customer support vs. engineering effort.

iudqnolq 3y ago

Under certain circumstances you might want to put backpressure on the email emitter. Under certain circumstances you might want to hard error. If I'm trying to reset my password and your UI says to check my email but I don't receive an email I'm gonna be annoyed. My first guess will be some kind of intermittent delivery issue, so I'll probably send a new request to your UI. If this is important to me then after an hour or so I might open a support ticket. I'll have wasted a bunch of time checking to see if I've finally got the email. I'd much rather have gotten an error and the retried when your status page indicates the issue is resolved.

lutzh 3y ago
Thanks for your feedback! Delayed e-mail can have of course have many other reasons than the service supposed to send it being down. But let's go with your example. Let's say you have a LoginService, and a MessagingService, and if you request a password reset, the LoginService sends a ResetRequested event. Now if the MessagingService is down, you'd still get told to check your inbox, although the e-mail hasn't even been sent. That's a bad user experience. Agreed. Your suggested solution is to make the communication between LoginService and MessagingService not event driven. So if the MessagingService is down, you can "hard error". But how much better is the experience then, really? As a user, you request the reset, and after a few seconds (you're already annoyed and checking if your internet connection is down), you get an error message that you can't reset your password right now. I'd argue that option is equally unacceptable. What you need to do, is to make the MessagingService better. In an event driven system, each participating service makes a promise. The MessagingService must give service level guarantees, it must be built and scaled to be able to process all the events it's listening to. If it has an error, not you should get the error message - you are the last person able to do something about it - but someone who fixes it. If you want a better user experience, don't build unreliable services and then apologize for the inconvenience. Built better (more reliable, more scalable) software. In the end, that's what event-driven architecture is about.
- iudqnolq 3y ago
  > Delayed e-mail can have of course have many other reasons than the service supposed to send it being down. Of course. But I was presuming that your email service makes an API call to an email service provider rather than speaking SMTP itself. That API call has three possible cases: - Ok: You're done, your ESP is responsible for SMTP retries. Maybe there's a webhook to re-enqueue something eventually after failures. - Transient error: You retry. But you probably don't want to wait that long and there's probably no point in interleaving attempts to send other emails through the same API. So you may as well "block" (as in yield to the async scheduler) while waiting to retry. - Persistent error: Here's when I suggest hard failing up to the user. > If it has an error, not you [the end user] should get the error message - you are the last person able to do something about it - but someone who fixes it. In the post you replied to I walked through in detail how my experience would be noticably better if I got an error message. > If you want a better user experience, don't build unreliable services and then apologize for the inconvenience. Built better (more reliable, more scalable) software. I agree you should attempt to eliminate all failability. I disagree that if you think your software is perfect you don't need to write code that provides the least unpleasant user experience possible if an error happens. Email is an exceptional example: it is theoretically impossible to write a service that infallibly delivers email to {gmail,yahoo,outlook} within a short period of time.
ericbarrett 3y ago
MessagingService includes its queue depth, or age, to LoginService in the enqueue response. If over $threshold, LoginService tells UI to add a parenthetical "(our email is backed up but you'll get it soon)" notice. Add alerting around high queue depth and I think you've buttoned this up.
- iudqnolq 3y ago
  Agree that's another way to solve it. Treating password reset emails as transient, erroring, and not enqueuing would also work and might be simpler.
  - ericbarrett 3y ago
    For sure, guess it depends on the volume of the service and balancing customer support vs. engineering effort.