34 ms·

I was bitten by this issue already two times. First time wth magic links via email. Suddenly your application depends on the mail infrastructure that sometimes

I was bitten by this issue already two times. First time wth magic links via email. Suddenly your application depends on the mail infrastructure that sometimes is hard to "scale on demand" or even isn't under your control. Second time with OTP via SMS. Both incidents led us to be unreachable for long periods of time, and the SLA provided by our mail provider and telecom weren't really known until the incident. We assumed that everything would just work under high load.

iKevinShah 6y ago
Quality insight from someone who has actually gone through this. Thank you! If you could share, can you share what exactly led to the discovery that things are unreachable. Were there any status check mechanisms you were able to put in place to check that incident was raised and / or solved? Also, do you still continue that (Email login / OTP login) OR is it moved something else?
- pelasaco 6y ago
  > If you could share, can you share what exactly led to the discovery that things are unreachable. We have some heavy monitoring with InfluxDB and Grafana built in our application and one of the alerts is if the number of logins/minute drop under a threshold. That's how we noticed it first time. The main reason was some network issues affecting our mail server provider. We were holding on a support ticket with our provider while trying to find a solution to our customers. After that we added as well we extended this monitoring to the mail queue. To be fair we used to monitor it before, but with the infra team, now we have it in our SRE dashboard, with Slack notifications and etc. > Also, do you still continue that (Email login / OTP login) OR is it moved something else? The OTP as second factor of authentication is something that we couldn't disable, but it's a requirement for one very specific application. We just looked for different partners with better SLA and built some monitoring around it. The Email login is still there, but we didn't roll-out it to all our applications as we initially intended to. We are still studying what would be the best solution here. The company is heavy user of microsoft's 365 mail service, and although the overall experience is pretty good, we have 0 influence in their SLA if we get impacted by any issue on their side. I don't think that the solution is bad per-se, just you have to plan mail infrastructure as core part of your application.

iKevinShah 6y ago

Quality insight from someone who has actually gone through this. Thank you! If you could share, can you share what exactly led to the discovery that things are unreachable. Were there any status check mechanisms you were able to put in place to check that incident was raised and / or solved? Also, do you still continue that (Email login / OTP login) OR is it moved something else?

pelasaco 6y ago
> If you could share, can you share what exactly led to the discovery that things are unreachable. We have some heavy monitoring with InfluxDB and Grafana built in our application and one of the alerts is if the number of logins/minute drop under a threshold. That's how we noticed it first time. The main reason was some network issues affecting our mail server provider. We were holding on a support ticket with our provider while trying to find a solution to our customers. After that we added as well we extended this monitoring to the mail queue. To be fair we used to monitor it before, but with the infra team, now we have it in our SRE dashboard, with Slack notifications and etc. > Also, do you still continue that (Email login / OTP login) OR is it moved something else? The OTP as second factor of authentication is something that we couldn't disable, but it's a requirement for one very specific application. We just looked for different partners with better SLA and built some monitoring around it. The Email login is still there, but we didn't roll-out it to all our applications as we initially intended to. We are still studying what would be the best solution here. The company is heavy user of microsoft's 365 mail service, and although the overall experience is pretty good, we have 0 influence in their SLA if we get impacted by any issue on their side. I don't think that the solution is bad per-se, just you have to plan mail infrastructure as core part of your application.