11 ms·

Mandrill has been down for over 30 hours with no explanation

290 points by slau 8y ago

slau 8y ago
Disclaimer: it's my tweet that is linked in this story, and I submitted it. There's no news story as far as I know. Better links are appreciated. Mandrill stopped delivering email at 04:51 UTC on the 4th of February. Nobody knows whether the received emails are lost, what the cause is, how soon any change will occur. It took Mandrill over 9 hours to acknowledge the issue. They are posting the same message over and over, and have now been silent for over 9 hours. Mandrill also recently got rid of their status page. A few months ago, their API has started returning nginx errors, and their status page looked like a Christmas tree; every reload would indicate different green/orange statuses. The certificate their MX servers uses is invalid (wrong domain), which prevents email delivery from compliant Danish email servers.
- rys 8y ago
  So it’s dead, then? At least those are not the signs of a healthy operation that just made a rectifiable mistake. Combined with the length of downtime, sounds really bad...
- adnans 8y ago
  We're currently using Mandrill to send outbound emails, although we've had issues with some API calls, all the outbound emails we sent have been delivered. Certain things like viewing email content look broken on certain emails which doesn't hurt the business, yet... It's true though that this service is nearing death as hasn't had an update to any of its features since it merge with Mailchimp. Looking to change service providers soon, as soon as we figure out how to render emails to html before sending. Mandrill has a `render` endpoint which makes this easy. None of the others have this yet.
  - slau 8y ago
    Yes, outbound has been solid for us, too. The issue is inbound. We literally have hundreds if not thousands of customer support agents unable to work because of this.
    - bvm 8y ago
      I've had lots of issues with outbound. Bulk sends failing completely. Individual seems to work but is very slow.
  - mcintyre1994 8y ago
    Not that it's a great alternative, but SendGrid uses Handlebars templates so it's pretty easy to render it yourself using any Handlebars library. You could probably quite easily write an AWS Lambda or similar that fetches a template using their API, populates it with data you post and returns the HTML.
  - brobdingnagians 8y ago
    The weirdest UX problem I've found with Mandrill is that you can't use the browser "CTRL-F" to search an email template unless the text is _currently_ visible. Not sure how they manged to mess up the search functionality for a text box that is intended to have massive amounts of text in it. It makes it harder to do a minor spot edit of a template than it should be. Things begin to make sense now...
    - crisscrosscrash 8y ago
      I believe this is a performance optimization to limit what's actually in the DOM for large documents. The code editor is fairly common and is also used by Google Tag Manager, but it annoys the heck out of me too.
  - losvedir 8y ago
    A few years ago when Mandrill dramatically increased prices there was a big exodus of HN users. I solicited pricing and transition info from a number of different providers and summarized it here[0]. [0] http://gabe.durazo.us/tech/hacker-news-mandrill-followup/ http://gabe.durazo.us/tech/hacker-news-mandrill-followup/
  - moltar 8y ago
    Take a look at MJML.
    - adnans 8y ago
      Already using MJML to design email templates and wrapping <mj-raw> tags around Mailchimp conditional tags like IF:SOMETHING .. |END:IF|. I guess it's time to render the html emails ourselves, merge with params and then use whatever other third party transactional email service.
  - ncarroll 8y ago
    Can't recommend Foundation for Emails highly enough for rendering html. I found it super easy to use (Moustache templates) and very flexible for managing multiple versions (translations) of a newsletter. The assignment I had was to write, coordinate translations, produce and send a quarterly newsletter with anywhere between 20 and 35 translations per issue + optional separate content to be included on a country-to-country basis. Production time from drafted newsletter to final send was about 2 weeks. The Foundation tool felt to me a little like jekyll for email. I had a a lot of fun with it. https://foundation.zurb.com/emails/docs/ https://foundation.zurb.com/emails/docs/ Disclaimer: I have no connection to Foundation. Just a delighted user.
  - zbentley 8y ago
    > Mandrill has a `render` endpoint which makes this easy. None of the others have this yet. Klaviyo provides similar functionality: https://www.klaviyo.com/docs/api/email-templates https://www.klaviyo.com/docs/api/email-templates Disclaimer: I work at Klaviyo.
  - kawsper 8y ago
    > all the outbound emails we sent have been delivered. You can't trust their interface during these outages, we have been there before, even though their Outbound page says "Delivered" it isn't unless there are SMTP events attached, if it looks like this, your email is queued, not sent: https://cdn.servnice.com/screenie/c1Og3TcrkLFV9Hg.jpg https://cdn.servnice.com/screenie/c1Og3TcrkLFV9Hg.jpg
  - than 8y ago
    We’ve been using https://MJML.io https://MJML.io and Handlebars and owning the rendering on our end. It’s been a breeze.
  - ryantownsend 8y ago
    > Mandrill has a `render` endpoint which makes this easy. None of the others have this yet. Campaign Monitor have the ability to take an API call and render into HTML using templates, this might do what you want: https://www.campaignmonitor.com/features/transactional-email/ https://www.campaignmonitor.com/features/transactional-email... (I'm unaffiliated with CM, just a customer)
- EwanToo 8y ago
  Sounds pretty rubbish for you and their other customers :( I'm afraid I don't have any answer for you, but I'm guessing quite a few people will be moving to sendgrid[1] - I'm not sure how I'd be able to trust a supplier after this lack of response to an outage. [1] - https://sendgrid.com/use-cases/transactional-email/ https://sendgrid.com/use-cases/transactional-email/
  - slau 8y ago
    We're actually moving to SES as we speak, and the move should be done in a few hours. It's something we had planned, but it just sucks that this is how a paid service gets terminated.
  - mcintyre1994 8y ago
    Recently moved our company from Mandrill to SendGrid - mainly because we were having deliverability issues with Mandrill. We're definitely a lot happier from that perspective (and obviously given this issue I'm glad we moved), and overall it's really good. A couple of things to be aware of: - Unlike Mandrill you can't send with a test API key and then view the emails. We end up sending real emails to a test inbox, which has some pros and cons - but cons include it costing email sends, and Gmail being awkward and silently de-duping sometimes. - You also can't view sent email content, at all (beyond the subject). Mandrill you can, Sendgrid don't seem to save it. - Mandrill/others probably has the same issue, but SendGrid without your own IP sends emails to new recipients extremely slowly. And they lock your whole account while they're doing it, password resets won't send to existing recipients and nor will test emails to your test inbox. - You also only get 3 days of email send history, you can pay for up to 30 I think. I'm not sure what the rule was with Mandrill but they seemed to keep a lot more available, though their search would time out most of the time so I'm not sure how much was really there and accessible.
    - bvm 8y ago
      Mandrill is 30 days of send history with 90 days of stats.
- Holybedd 8y ago
  > which prevents email delivery from compliant Danish email servers. Any email server which validates that the remote host name matches the cert will be be very sad. Doesn't matter if it's compliant (to what btw?), in practice email server owners are sloppy.
  - ahje 8y ago
    The Danish authorities have apparently mandated TLS 1.2 when sending certain mails, due to their interpretation of GDPR. I don't have the details though.
samat 8y ago
I don't care about Mandrill (who uses it, anyway), but deeply concerned if I should expect same quality, incident response and attitude to customers from parent company — Mailchimp.
- gfwhukku 8y ago
  That's what I was thinking. Who hasn't moved to Mailgun already?
  - Forge36 8y ago
    I just setup an account with mailgun. Pleasantly surprised at how easy it was.
  - samat 8y ago
    Why not AWS SES?
    - frereubu 8y ago
      AWS SES doesn't have things like reporting, bounce handling etc. out of the box - you need to set up all of that yourself (or at least you did when I last looked at it a few years ago).
      - ethagnawl 8y ago
        This was all still true ~6 months ago. Compared to Mailchimp and friends, SES is _very_ low level. The other things I found really frustrating about SES were: templates had to be defined inline in a JSON file and then sent to SES via the AWS CLI. So, since there's is no online/visual editor, copy changes and the like required a developer to rebuild/sanitize/minify the template source and then update it via the AWS CLI. It also took _way_ longer than it should have to have our rate limit bumped up to a reasonable level. IIRC, it took ~one week for my request to be processed (after submitting proof that we owned the domain, etc.) and it was only after a fit on twitter that AWS Support followed up with me and escalated the issue.
a2tech 8y ago
We moved over to SES a long time ago. Mandrill was basically the cheapest game in town, but their terrible service coupled with Mailchimp essentially abandoning the platform after their acquisition made us jump ship. I feel bad for anyone still using their service.
- throwaway2016a 8y ago
  Depends on your volume. Mandril has a monthly fee and thanks to pay as you go pricing SES is cheaper up to a decent volume for a lot of startups sending just transactional email.
jdc0589 8y ago
> MOST outbound emails are sending oh, so normal operation then
- jdc0589 8y ago
  for context: here's a partial view of our outbound error rate over the past 60 hours (this is maybe ~30% of all total mandrill errors): https://puu.sh/CHDxb/729e52451f.png https://puu.sh/CHDxb/729e52451f.png. We see around a 2% failure rate in Mandrill requests on a normal day though, its bad enough we do two immediate retries before allowing our queuing system to start handling retries with backoff.
ericcholis 8y ago
Things like this influence stakeholder decisions down the road. I've been evaluating migrating to Mandrill for some time to give our designer more control over transactional emails. Now, it's unlikely that I'll do so. (Worth noting, that I'm a happy long-time Mailchimp customer)
- shedside 8y ago
  We used Mandrill for some time, then moved to SendGrid, and just recently have moved again to Postmark because we were seeing delivery issues. FWIW, Postmark has been fantastic.
  - pbowyer 8y ago
    Postmark is awesome!
  - brandonmenc 8y ago
    Postmark is the best. Rock solid, and a singular focus on one thing. I've been using it for years.
frereubu 8y ago
There seem to be quite a few snarky comments about Mandrill here, but we've been using it for a few years and until today we've been happy with the service. I'd be interested to hear concrete reasons (apart from the current outage!) why Mandrill isn't as good as other services for a use case where I want reporting, bounce handling etc. as part of the service (e.g. not AWS SES).
- tyingq 8y ago
  This one seems constructive, concerning, outlines events prior to today, and was posted before your post: "Mandrill also recently got rid of their status page. A few months ago, their API has started returning nginx errors, and their status page looked like a Christmas tree; every reload would indicate different green/orange statuses. The certificate their MX servers uses is invalid (wrong domain), which prevents email delivery from compliant Danish email servers."
  - frereubu 8y ago
    That's true. None of those affect the way I use Mandrill, which is perhaps why I didn't pay too much attention to it, but it doesn't paint a good picture for sure.
neya 8y ago
Unfortunately this is standard MailChimp way of doing things ever since they screwed over paying customers and merged with Mandrill [1]. They are the opposite of a transparent organization - they are the GoDaddy of this business. (I was super pissed once they removed their status page..WTF). We hit these errors a few months ago with our clients and I put up a roadmap to move all of them off the terrible joke that Mandrill is, in under a month. None of my clients regretted the move. I really don't miss anything about them and the market has caught up and they've been stagnant for way too long without any changes whatsoever and are still charging a premium for a prototype level quality of service. If you're still using Mandrill in 2019, move out as soon as you can. I'm saying this with public interest as I had enough of these jokers. [1] http://www.dangrossman.info/2016/02/28/mandrills-betrayal/ http://www.dangrossman.info/2016/02/28/mandrills-betrayal/ Edit: I built a Mandrill clone built on SES, Phoenix/Elixir and hosted on Google AppEngine which is basically Google's managed hosting service. I moved everything there and it works pretty well, planning to open source it at some point. The awesome thing about this is I can work on some specific features my clients want that isn't available on vanilla Mandrill. I chose this route because I lost a lot of money with Mandrill compensating my clients for their fault, so I calculated that the money I lost over time would have been the same amount if I had invested in building a Mandrill clone myself. I plan to open source it at some point.
- buf 8y ago
  I moved to sendgrid when they announced mandrill's merger and have never been happier.
  - cremp 8y ago
    Except now, sendgrid is merged with twilio; so... history repeating?
    - edoceo 8y ago
      No. Twilio has a status page :)
      - 8y ago
jrockway 8y ago
We use Mandrill to get inbound emails into our support ticketing system. This incident has pushed me in the direction of switching to ZenDesk or something... but according to their status page, they also lost inbound emails yesterday. Coincidence or is it Mandrill all the way down?
- f3r3nc 8y ago
  We have issues with both. Some emails do go out, some not. No inbound.
kawsper 8y ago
Just remember that you can't trust their interface, even though their Outbound page says "Delivered" it isn't delivered unless there are SMTP events attached, if it looks like this, your email is queued, not sent: https://cdn.servnice.com/screenie/c1Og3TcrkLFV9Hg.jpg https://cdn.servnice.com/screenie/c1Og3TcrkLFV9Hg.jpg
- chrismeller 8y ago
  Wait, what? So delivered doesn’t mean delivered... who thought this was acceptable UX?
- gondo 8y ago
  exactly! this was the main reason I moved away.
entity345 8y ago
Oh boy, that twitter thread is brutal... It looks like Mandrill's staff has gone MIA.
bvm 8y ago
This is honestly the most poorly communicated downtime I've ever experienced. Now, they give a non-update saying they're going to send an email (yes, email) about it later today: https://twitter.com/mandrillapp/status/1092810757929086977 https://twitter.com/mandrillapp/status/1092810757929086977
Wouter33 8y ago
Mailchimp just sent out an update per mail: https://pastebin.com/6TN10AZB https://pastebin.com/6TN10AZB TLDR: One of their five Postgres clusters went into read-only mode due to an Transaction ID Wraparound issue. Restoring this can take up to several days (!).
GuyPostington 8y ago
Got this email just now. - - - - Hello, We’re contacting you about an ongoing outage with the Mandrill app. This email provides background on what happened and how users are affected, what we’re doing to address the issue, and what’s next for our customers. What happened Mandrill uses a sharded Postgres setup as one of our main datastores. On Sunday, February 3, at 10:30pm EST, 1 of our 5 physical Postgres instances saw a significant spike in writes. The spike in writes triggered a Transaction ID Wraparound issue. When this occurs, database activity is completely halted. The database sets itself in read-only mode until offline maintenance (known as vacuuming) can occur. The database is large—running the vacuum process takes a significant amount of time and resources, and there’s no clear way to track progress. Customer impact The impact to users could come in the form of not tracking opens, clicks, bounces, email sends, inbound email, webhook events, and more. Right now, it looks like the database outage is affecting up to 20% of our outbound volume as well as a majority of inbound email and webhooks. What we’re doing to address this We don’t have an estimated time for when the vacuum process and cleanup work will be complete. While we have a parallel set of tasks going to try to get the database back in working order, these efforts are also slow and difficult with a database of this size. We’re trying everything we can to finish this process as quickly as possible, but this could take several days, or longer. We hope to have more information and a timeline for resolution soon. In the meantime, it’s possible that you may see errors related to sending and receiving emails. We’ll continue to update you on our progress by email and let you know as soon as these issues are fully resolved. What’s next We apologize for the disruption to your business. Once the outage is resolved, we plan to offer refunds to all affected users. You don’t need to take any action at this time—we’ll share details in a follow-up email and will automatically credit your account. Again, we’re sorry for the interruption and we hope to have good news to share soon.
- hartator 8y ago
  Ha! For once it’s not MongoDB but Postgres. I wonder why the sending is effected though. Can’t they run their service with an empty databse in the meantime?
- jmagoon 8y ago
  Why does one shard being down remove all their inbound functionality? I'm struggling to understand the purpose of sharding if you can't pull a node offline and replace it while you deal with the wraparound. Is it part of postgres that if one shard has an issue, the entire cluster goes into read only mode?
lgleason 8y ago
I live in Atlanta where MailChimp (the parent company) is headquartered. They hire for progressive politics first, actual technical skills second/third. Many of their employees spend a large part of their day on social justice crusades targeting community members for wrong-think instead of gasp actually building a better products and taking care of customers. They are also known to remove accounts on the system based on these partisan politics. At some point this is bound to catch up to them as competitors actually invest in their product. In the meantime, they will try to advertise the product like crazy and enough people will buy it without actually researching their actual customer service until this void is filled etc..
- tomnipotent 8y ago
  > They are also known to remove accounts on the system based on these partisan politics. Please provide proof when making claims like this. Infowars/Alex Jones doesn't count, they were universally blacklisted.
  - retrogradeorbit 8y ago
    ROFL. Please provide proof they remove accounts based on partisan politics! Except for that case that they removed an account based on partisan politics!
    - tomnipotent 8y ago
      And you think an isolated incident somehow proves a trend or is otherwise concrete evidence of repeat violations? There's zero evidence of a systemic problem.
    - deleted 8y ago
      [deleted]
coleca 8y ago
Sending #HugOps over to the Mandrill Ops team. As frustrating it is to not have a service you depend on be available, if you've been around long enough, you've probably been on the other end of that kind of outage and know it isn't fun, expected or malicious.
olegd1280 8y ago
ESP's like mailchimp, constant contact, and other always have some issues with emails...mainly because of the contacts list.. I made the switch to mailclickconvert.com from constant conatct and its been much better.
niftylettuce 8y ago
If you need a service to use for email forwarding, try https://forwardemail.net https://forwardemail.net Source Code: https://github.com/niftylettuce/forward-email https://github.com/niftylettuce/forward-email
ceejayoz 8y ago
We moved off Mandrill back in 2016 when they changed their TOS effective immediately for one of their major use cases (sending newsletters). I'm kinda glad now they forced the issue. HN coverage of that event: https://news.ycombinator.com/item?id=11203056 https://news.ycombinator.com/item?id=11203056
Dphilman 8y ago
Wow, that's pretty bad. I'm sure a lot of companies were affected. No ETA on a fix is really scary. We currently use Law Ruler to send out our emails and have had no issues in the last 2 years. If anyone needs a new solution fast I would reach out to their support ( form on website ).
Animats 8y ago
It's a spamming service, right? (Not, as one might think from the name, a gay dating service.) Is it a big deal if it's down to anyone but the spammers? There's an annoying tendency to combine spam services with actual transaction reports. (A transaction report is "Your order has shipped and here is the tracking number", not "We have a new product") Because everybody blocks the spammers. Constant Contact went down that road, and now you can't reliably use Constant Contact for transactions.
bevacqua 8y ago
Thrilled that I moved to mailgun when they announced the merge of Mandrill into MailChimp
etjossem 8y ago
A few of my teammates at SendGrid have been following the situation, and we definitely feel for the engineers who are scrambling to fix the problem. It's never fun to get paged, especially when the trust of your customers on the line. Some folks on the email thread were personally involved in handling major outages in the early days. We've had to learn a lot of hard lessons since then. Even when everything seems like it's going fine ("wow, we're growing so fast, good problems!"), scaling issues could be right around the corner. Anyone with a large enough installation of Postgres could've had the wraparound issue we're seeing right now. That's why it's important to monitor for what could go wrong, detect these issues early, and provide customers with rapid communication so they can plan around it. Sending our best wishes to the MailChimp engineering teams working on the problem right now. Good luck, you've got this!
ian0 8y ago
The email from support (only sent a few hours ago) stated: > The impact to users could come in the form of not tracking opens, clicks, bounces, email sends, inbound email, webhook events, and more. Right now, it looks like the database outage is affecting up to 20% of our outbound volume as well as a majority of inbound email and webhooks. It makes it seem like the actual sending emails were not effected, just "tracking". I landed on this thread as some reporting emails of ours weren't sent. Can anyone confirm it effected the sending of mail too? Luckily we had nothing critical running through Mandrill, but I feel sorry for those who did, given it hit right around CNY where many people will be on holidays.
bks 8y ago
If you just need outbound email submitted via smtp try outboundsmtp.com (I run it)

290 points by slau 8y ago

slau 8y ago

Disclaimer: it's my tweet that is linked in this story, and I submitted it. There's no news story as far as I know. Better links are appreciated. Mandrill stopped delivering email at 04:51 UTC on the 4th of February. Nobody knows whether the received emails are lost, what the cause is, how soon any change will occur. It took Mandrill over 9 hours to acknowledge the issue. They are posting the same message over and over, and have now been silent for over 9 hours. Mandrill also recently got rid of their status page. A few months ago, their API has started returning nginx errors, and their status page looked like a Christmas tree; every reload would indicate different green/orange statuses. The certificate their MX servers uses is invalid (wrong domain), which prevents email delivery from compliant Danish email servers.

rys 8y ago
So it’s dead, then? At least those are not the signs of a healthy operation that just made a rectifiable mistake. Combined with the length of downtime, sounds really bad...
adnans 8y ago
We're currently using Mandrill to send outbound emails, although we've had issues with some API calls, all the outbound emails we sent have been delivered. Certain things like viewing email content look broken on certain emails which doesn't hurt the business, yet... It's true though that this service is nearing death as hasn't had an update to any of its features since it merge with Mailchimp. Looking to change service providers soon, as soon as we figure out how to render emails to html before sending. Mandrill has a `render` endpoint which makes this easy. None of the others have this yet.
- slau 8y ago
  Yes, outbound has been solid for us, too. The issue is inbound. We literally have hundreds if not thousands of customer support agents unable to work because of this.
  - bvm 8y ago
    I've had lots of issues with outbound. Bulk sends failing completely. Individual seems to work but is very slow.
- mcintyre1994 8y ago
  Not that it's a great alternative, but SendGrid uses Handlebars templates so it's pretty easy to render it yourself using any Handlebars library. You could probably quite easily write an AWS Lambda or similar that fetches a template using their API, populates it with data you post and returns the HTML.
- brobdingnagians 8y ago
  The weirdest UX problem I've found with Mandrill is that you can't use the browser "CTRL-F" to search an email template unless the text is _currently_ visible. Not sure how they manged to mess up the search functionality for a text box that is intended to have massive amounts of text in it. It makes it harder to do a minor spot edit of a template than it should be. Things begin to make sense now...
  - crisscrosscrash 8y ago
    I believe this is a performance optimization to limit what's actually in the DOM for large documents. The code editor is fairly common and is also used by Google Tag Manager, but it annoys the heck out of me too.
- losvedir 8y ago
  A few years ago when Mandrill dramatically increased prices there was a big exodus of HN users. I solicited pricing and transition info from a number of different providers and summarized it here[0]. [0] http://gabe.durazo.us/tech/hacker-news-mandrill-followup/ http://gabe.durazo.us/tech/hacker-news-mandrill-followup/
- moltar 8y ago
  Take a look at MJML.
  - adnans 8y ago
    Already using MJML to design email templates and wrapping <mj-raw> tags around Mailchimp conditional tags like IF:SOMETHING .. |END:IF|. I guess it's time to render the html emails ourselves, merge with params and then use whatever other third party transactional email service.
- ncarroll 8y ago
  Can't recommend Foundation for Emails highly enough for rendering html. I found it super easy to use (Moustache templates) and very flexible for managing multiple versions (translations) of a newsletter. The assignment I had was to write, coordinate translations, produce and send a quarterly newsletter with anywhere between 20 and 35 translations per issue + optional separate content to be included on a country-to-country basis. Production time from drafted newsletter to final send was about 2 weeks. The Foundation tool felt to me a little like jekyll for email. I had a a lot of fun with it. https://foundation.zurb.com/emails/docs/ https://foundation.zurb.com/emails/docs/ Disclaimer: I have no connection to Foundation. Just a delighted user.
- zbentley 8y ago
  > Mandrill has a `render` endpoint which makes this easy. None of the others have this yet. Klaviyo provides similar functionality: https://www.klaviyo.com/docs/api/email-templates https://www.klaviyo.com/docs/api/email-templates Disclaimer: I work at Klaviyo.
- kawsper 8y ago
  > all the outbound emails we sent have been delivered. You can't trust their interface during these outages, we have been there before, even though their Outbound page says "Delivered" it isn't unless there are SMTP events attached, if it looks like this, your email is queued, not sent: https://cdn.servnice.com/screenie/c1Og3TcrkLFV9Hg.jpg https://cdn.servnice.com/screenie/c1Og3TcrkLFV9Hg.jpg
- than 8y ago
  We’ve been using https://MJML.io https://MJML.io and Handlebars and owning the rendering on our end. It’s been a breeze.
- ryantownsend 8y ago
  > Mandrill has a `render` endpoint which makes this easy. None of the others have this yet. Campaign Monitor have the ability to take an API call and render into HTML using templates, this might do what you want: https://www.campaignmonitor.com/features/transactional-email/ https://www.campaignmonitor.com/features/transactional-email... (I'm unaffiliated with CM, just a customer)
EwanToo 8y ago
Sounds pretty rubbish for you and their other customers :( I'm afraid I don't have any answer for you, but I'm guessing quite a few people will be moving to sendgrid[1] - I'm not sure how I'd be able to trust a supplier after this lack of response to an outage. [1] - https://sendgrid.com/use-cases/transactional-email/ https://sendgrid.com/use-cases/transactional-email/
- slau 8y ago
  We're actually moving to SES as we speak, and the move should be done in a few hours. It's something we had planned, but it just sucks that this is how a paid service gets terminated.
- mcintyre1994 8y ago
  Recently moved our company from Mandrill to SendGrid - mainly because we were having deliverability issues with Mandrill. We're definitely a lot happier from that perspective (and obviously given this issue I'm glad we moved), and overall it's really good. A couple of things to be aware of: - Unlike Mandrill you can't send with a test API key and then view the emails. We end up sending real emails to a test inbox, which has some pros and cons - but cons include it costing email sends, and Gmail being awkward and silently de-duping sometimes. - You also can't view sent email content, at all (beyond the subject). Mandrill you can, Sendgrid don't seem to save it. - Mandrill/others probably has the same issue, but SendGrid without your own IP sends emails to new recipients extremely slowly. And they lock your whole account while they're doing it, password resets won't send to existing recipients and nor will test emails to your test inbox. - You also only get 3 days of email send history, you can pay for up to 30 I think. I'm not sure what the rule was with Mandrill but they seemed to keep a lot more available, though their search would time out most of the time so I'm not sure how much was really there and accessible.
  - bvm 8y ago
    Mandrill is 30 days of send history with 90 days of stats.
Holybedd 8y ago
> which prevents email delivery from compliant Danish email servers. Any email server which validates that the remote host name matches the cert will be be very sad. Doesn't matter if it's compliant (to what btw?), in practice email server owners are sloppy.
- ahje 8y ago
  The Danish authorities have apparently mandated TLS 1.2 when sending certain mails, due to their interpretation of GDPR. I don't have the details though.

samat 8y ago

I don't care about Mandrill (who uses it, anyway), but deeply concerned if I should expect same quality, incident response and attitude to customers from parent company — Mailchimp.

gfwhukku 8y ago
That's what I was thinking. Who hasn't moved to Mailgun already?
- Forge36 8y ago
  I just setup an account with mailgun. Pleasantly surprised at how easy it was.
- samat 8y ago
  Why not AWS SES?
  - frereubu 8y ago
    AWS SES doesn't have things like reporting, bounce handling etc. out of the box - you need to set up all of that yourself (or at least you did when I last looked at it a few years ago).
    - ethagnawl 8y ago
      This was all still true ~6 months ago. Compared to Mailchimp and friends, SES is _very_ low level. The other things I found really frustrating about SES were: templates had to be defined inline in a JSON file and then sent to SES via the AWS CLI. So, since there's is no online/visual editor, copy changes and the like required a developer to rebuild/sanitize/minify the template source and then update it via the AWS CLI. It also took _way_ longer than it should have to have our rate limit bumped up to a reasonable level. IIRC, it took ~one week for my request to be processed (after submitting proof that we owned the domain, etc.) and it was only after a fit on twitter that AWS Support followed up with me and escalated the issue.

a2tech 8y ago

We moved over to SES a long time ago. Mandrill was basically the cheapest game in town, but their terrible service coupled with Mailchimp essentially abandoning the platform after their acquisition made us jump ship. I feel bad for anyone still using their service.

throwaway2016a 8y ago
Depends on your volume. Mandril has a monthly fee and thanks to pay as you go pricing SES is cheaper up to a decent volume for a lot of startups sending just transactional email.

jdc0589 8y ago

> MOST outbound emails are sending oh, so normal operation then

jdc0589 8y ago
for context: here's a partial view of our outbound error rate over the past 60 hours (this is maybe ~30% of all total mandrill errors): https://puu.sh/CHDxb/729e52451f.png https://puu.sh/CHDxb/729e52451f.png. We see around a 2% failure rate in Mandrill requests on a normal day though, its bad enough we do two immediate retries before allowing our queuing system to start handling retries with backoff.

ericcholis 8y ago

Things like this influence stakeholder decisions down the road. I've been evaluating migrating to Mandrill for some time to give our designer more control over transactional emails. Now, it's unlikely that I'll do so. (Worth noting, that I'm a happy long-time Mailchimp customer)

shedside 8y ago
We used Mandrill for some time, then moved to SendGrid, and just recently have moved again to Postmark because we were seeing delivery issues. FWIW, Postmark has been fantastic.
- pbowyer 8y ago
  Postmark is awesome!
- brandonmenc 8y ago
  Postmark is the best. Rock solid, and a singular focus on one thing. I've been using it for years.

frereubu 8y ago

There seem to be quite a few snarky comments about Mandrill here, but we've been using it for a few years and until today we've been happy with the service. I'd be interested to hear concrete reasons (apart from the current outage!) why Mandrill isn't as good as other services for a use case where I want reporting, bounce handling etc. as part of the service (e.g. not AWS SES).

tyingq 8y ago
This one seems constructive, concerning, outlines events prior to today, and was posted before your post: "Mandrill also recently got rid of their status page. A few months ago, their API has started returning nginx errors, and their status page looked like a Christmas tree; every reload would indicate different green/orange statuses. The certificate their MX servers uses is invalid (wrong domain), which prevents email delivery from compliant Danish email servers."
- frereubu 8y ago
  That's true. None of those affect the way I use Mandrill, which is perhaps why I didn't pay too much attention to it, but it doesn't paint a good picture for sure.

neya 8y ago

Unfortunately this is standard MailChimp way of doing things ever since they screwed over paying customers and merged with Mandrill [1]. They are the opposite of a transparent organization - they are the GoDaddy of this business. (I was super pissed once they removed their status page..WTF). We hit these errors a few months ago with our clients and I put up a roadmap to move all of them off the terrible joke that Mandrill is, in under a month. None of my clients regretted the move. I really don't miss anything about them and the market has caught up and they've been stagnant for way too long without any changes whatsoever and are still charging a premium for a prototype level quality of service. If you're still using Mandrill in 2019, move out as soon as you can. I'm saying this with public interest as I had enough of these jokers. [1] http://www.dangrossman.info/2016/02/28/mandrills-betrayal/ http://www.dangrossman.info/2016/02/28/mandrills-betrayal/ Edit: I built a Mandrill clone built on SES, Phoenix/Elixir and hosted on Google AppEngine which is basically Google's managed hosting service. I moved everything there and it works pretty well, planning to open source it at some point. The awesome thing about this is I can work on some specific features my clients want that isn't available on vanilla Mandrill. I chose this route because I lost a lot of money with Mandrill compensating my clients for their fault, so I calculated that the money I lost over time would have been the same amount if I had invested in building a Mandrill clone myself. I plan to open source it at some point.

buf 8y ago
I moved to sendgrid when they announced mandrill's merger and have never been happier.
- cremp 8y ago
  Except now, sendgrid is merged with twilio; so... history repeating?
  - edoceo 8y ago
    No. Twilio has a status page :)
    - 8y ago

jrockway 8y ago

We use Mandrill to get inbound emails into our support ticketing system. This incident has pushed me in the direction of switching to ZenDesk or something... but according to their status page, they also lost inbound emails yesterday. Coincidence or is it Mandrill all the way down?

f3r3nc 8y ago
We have issues with both. Some emails do go out, some not. No inbound.

kawsper 8y ago

Just remember that you can't trust their interface, even though their Outbound page says "Delivered" it isn't delivered unless there are SMTP events attached, if it looks like this, your email is queued, not sent: https://cdn.servnice.com/screenie/c1Og3TcrkLFV9Hg.jpg https://cdn.servnice.com/screenie/c1Og3TcrkLFV9Hg.jpg

chrismeller 8y ago
Wait, what? So delivered doesn’t mean delivered... who thought this was acceptable UX?
gondo 8y ago
exactly! this was the main reason I moved away.

entity345 8y ago

Oh boy, that twitter thread is brutal... It looks like Mandrill's staff has gone MIA.

bvm 8y ago

This is honestly the most poorly communicated downtime I've ever experienced. Now, they give a non-update saying they're going to send an email (yes, email) about it later today: https://twitter.com/mandrillapp/status/1092810757929086977 https://twitter.com/mandrillapp/status/1092810757929086977

Wouter33 8y ago

Mailchimp just sent out an update per mail: https://pastebin.com/6TN10AZB https://pastebin.com/6TN10AZB TLDR: One of their five Postgres clusters went into read-only mode due to an Transaction ID Wraparound issue. Restoring this can take up to several days (!).

GuyPostington 8y ago

Got this email just now. - - - - Hello, We’re contacting you about an ongoing outage with the Mandrill app. This email provides background on what happened and how users are affected, what we’re doing to address the issue, and what’s next for our customers. What happened Mandrill uses a sharded Postgres setup as one of our main datastores. On Sunday, February 3, at 10:30pm EST, 1 of our 5 physical Postgres instances saw a significant spike in writes. The spike in writes triggered a Transaction ID Wraparound issue. When this occurs, database activity is completely halted. The database sets itself in read-only mode until offline maintenance (known as vacuuming) can occur. The database is large—running the vacuum process takes a significant amount of time and resources, and there’s no clear way to track progress. Customer impact The impact to users could come in the form of not tracking opens, clicks, bounces, email sends, inbound email, webhook events, and more. Right now, it looks like the database outage is affecting up to 20% of our outbound volume as well as a majority of inbound email and webhooks. What we’re doing to address this We don’t have an estimated time for when the vacuum process and cleanup work will be complete. While we have a parallel set of tasks going to try to get the database back in working order, these efforts are also slow and difficult with a database of this size. We’re trying everything we can to finish this process as quickly as possible, but this could take several days, or longer. We hope to have more information and a timeline for resolution soon. In the meantime, it’s possible that you may see errors related to sending and receiving emails. We’ll continue to update you on our progress by email and let you know as soon as these issues are fully resolved. What’s next We apologize for the disruption to your business. Once the outage is resolved, we plan to offer refunds to all affected users. You don’t need to take any action at this time—we’ll share details in a follow-up email and will automatically credit your account. Again, we’re sorry for the interruption and we hope to have good news to share soon.

hartator 8y ago
Ha! For once it’s not MongoDB but Postgres. I wonder why the sending is effected though. Can’t they run their service with an empty databse in the meantime?
jmagoon 8y ago
Why does one shard being down remove all their inbound functionality? I'm struggling to understand the purpose of sharding if you can't pull a node offline and replace it while you deal with the wraparound. Is it part of postgres that if one shard has an issue, the entire cluster goes into read only mode?

lgleason 8y ago

I live in Atlanta where MailChimp (the parent company) is headquartered. They hire for progressive politics first, actual technical skills second/third. Many of their employees spend a large part of their day on social justice crusades targeting community members for wrong-think instead of gasp actually building a better products and taking care of customers. They are also known to remove accounts on the system based on these partisan politics. At some point this is bound to catch up to them as competitors actually invest in their product. In the meantime, they will try to advertise the product like crazy and enough people will buy it without actually researching their actual customer service until this void is filled etc..

tomnipotent 8y ago
> They are also known to remove accounts on the system based on these partisan politics. Please provide proof when making claims like this. Infowars/Alex Jones doesn't count, they were universally blacklisted.
- retrogradeorbit 8y ago
  ROFL. Please provide proof they remove accounts based on partisan politics! Except for that case that they removed an account based on partisan politics!
  - tomnipotent 8y ago
    And you think an isolated incident somehow proves a trend or is otherwise concrete evidence of repeat violations? There's zero evidence of a systemic problem.
  - deleted 8y ago
    [deleted]

coleca 8y ago

Sending #HugOps over to the Mandrill Ops team. As frustrating it is to not have a service you depend on be available, if you've been around long enough, you've probably been on the other end of that kind of outage and know it isn't fun, expected or malicious.

olegd1280 8y ago

ESP's like mailchimp, constant contact, and other always have some issues with emails...mainly because of the contacts list.. I made the switch to mailclickconvert.com from constant conatct and its been much better.

niftylettuce 8y ago

If you need a service to use for email forwarding, try https://forwardemail.net https://forwardemail.net Source Code: https://github.com/niftylettuce/forward-email https://github.com/niftylettuce/forward-email

ceejayoz 8y ago

We moved off Mandrill back in 2016 when they changed their TOS effective immediately for one of their major use cases (sending newsletters). I'm kinda glad now they forced the issue. HN coverage of that event: https://news.ycombinator.com/item?id=11203056 https://news.ycombinator.com/item?id=11203056

Dphilman 8y ago

Wow, that's pretty bad. I'm sure a lot of companies were affected. No ETA on a fix is really scary. We currently use Law Ruler to send out our emails and have had no issues in the last 2 years. If anyone needs a new solution fast I would reach out to their support ( form on website ).

Animats 8y ago

It's a spamming service, right? (Not, as one might think from the name, a gay dating service.) Is it a big deal if it's down to anyone but the spammers? There's an annoying tendency to combine spam services with actual transaction reports. (A transaction report is "Your order has shipped and here is the tracking number", not "We have a new product") Because everybody blocks the spammers. Constant Contact went down that road, and now you can't reliably use Constant Contact for transactions.

bevacqua 8y ago

Thrilled that I moved to mailgun when they announced the merge of Mandrill into MailChimp

etjossem 8y ago

A few of my teammates at SendGrid have been following the situation, and we definitely feel for the engineers who are scrambling to fix the problem. It's never fun to get paged, especially when the trust of your customers on the line. Some folks on the email thread were personally involved in handling major outages in the early days. We've had to learn a lot of hard lessons since then. Even when everything seems like it's going fine ("wow, we're growing so fast, good problems!"), scaling issues could be right around the corner. Anyone with a large enough installation of Postgres could've had the wraparound issue we're seeing right now. That's why it's important to monitor for what could go wrong, detect these issues early, and provide customers with rapid communication so they can plan around it. Sending our best wishes to the MailChimp engineering teams working on the problem right now. Good luck, you've got this!

ian0 8y ago

The email from support (only sent a few hours ago) stated: > The impact to users could come in the form of not tracking opens, clicks, bounces, email sends, inbound email, webhook events, and more. Right now, it looks like the database outage is affecting up to 20% of our outbound volume as well as a majority of inbound email and webhooks. It makes it seem like the actual sending emails were not effected, just "tracking". I landed on this thread as some reporting emails of ours weren't sent. Can anyone confirm it effected the sending of mail too? Luckily we had nothing critical running through Mandrill, but I feel sorry for those who did, given it hit right around CNY where many people will be on holidays.

bks 8y ago

If you just need outbound email submitted via smtp try outboundsmtp.com (I run it)