9 ms·

Impact to DigitalOcean customers resulting from Mailchimp security incident

168 points by smitop 4y ago

dynamohk 4y ago
You would think Digital Ocean being a cloud hosting provider they could do email themselves, secure customer data and not provide it to third parties. Great write up though.
- clintonb 4y ago
  I would not think that because cloud hosting and email delivery are very different problems to solve.
shawncampbell 4y ago
> on August 8th, our Security Operations team was made aware of a customer who claimed their password had been reset, without their initiation. > One of the first discoveries was a non-DigitalOcean email address that appeared on a regular email from Mailchimp on August 7th. > Soon after we discovered an issue with our Mailchimp account on August 8th, we initiated contact with Mailchimp, both via traditional support channels and other escalation methods. On August 10th, we had our first actionable response
- ben_jones 4y ago
  > both via traditional support channels and other escalation methods I wish our industry could just be honest that many meaningful escalations have to happen via discriminatory back-channeling, contacting friends, family, former co-workers, ANYONE who might have an in with the organization. Its discriminatory because if you don't know someone you can be SOL.
  - xtracto 4y ago
    Mailchimp/Mandrill have specially bad customer support and business user experience. At one point they closed the account of a company I was working at without any previous notice and without any recourse. They just sent a blanket email "Your account has been suspended". They left us scrambling for an alternative (we had both transactional and marketing email). We happily migrated to SES for transactional email and SendGrid for marketing. I no longer recommend or use Mailchimp.
    - 44gg44gg 4y ago
      My issue with SES is that your account may be placed in a Sandbox with no explanation other than to delete your account.
      - Godel_unicode 4y ago
        This is because of fraud. Everyone either does this, or enables the fraudsters. The only third option is to get rid of email, except the fraudsters would just move on to the next thing. Assume that you’ll have availability problems with email and engineer with that in mind.
        capableweb 4y ago
        Claiming that you either need to ban users without recurse or suffer from enabling fraudsters is why people start feeling like corporations don't have any human employees anymore. If you care about your customers, you could absolutely reach out and verify if the user you are about to/have blocked is a fraudster or not, and then act accordingly. The reason most just say "You're banned, bye" is because they don't want to do that work, and subsequently don't care about their users one bit.
        Godel_unicode 4y ago
        They ban thousands of accounts per day. Every day. If you can solve this problem you will have discovered a license to print money, so by all means try. But the problem is much harder than you are making it sound.
        capableweb 4y ago
        On the other hand, how many do you think would reach out when they get blocked? Most actual fraudsters wouldn't. Of those that do, you can take a look at what they are sending and quickly see if they are actually doing fraud or not. I could say the same thing to you, you're making it sound harder than what it is. But if you're set to the mindset of saving money, I understand minimising work makes sense.
  - preommr 4y ago
    > Its discriminatory because if you don't know someone you can be SOL. Thanks for explicitly explaining that part. Now that you've put it that way, it's a pretty apt term but I probably wouldn't have been able to figure out because that term is usually used for things like gender, race, etc. and my brain immediately jumped to that. Maybe I am just stupid, but sometimes it shouldn't be that hard to understand what people mean and it helps to be more explicit.
    - eastbound 4y ago
      Good opportunity to talk about it. An awful lot of people are extremely solitary. I’d wager this is more of a problem than racial discrimination because: - Lonely people aren’t visible, by definition, - Solitude increases racism, so it would be worth solving, - They often end up creating companies and being one’s boss.
xtracto 4y ago
Aaah, so THAT's why their email verification was not working for several hours [1] [2]! [1] https://news.ycombinator.com/item?id=32398773 https://news.ycombinator.com/item?id=32398773 [2] https://news.ycombinator.com/item?id=32392935 https://news.ycombinator.com/item?id=32392935
nerdawson 4y ago
It took two days for a company the size of DO to get an actionable response. What hope do the rest of us have? Interesting write up and 2FA by default sounds like a sensible move. If you’re the type of user to have a DO account, you should be perfectly capable of using 2FA.
- throwntoday 4y ago
  MFA is inconvenient
  - deleted 4y ago
    [deleted]
  - teeray 4y ago
    Especially when you generate one-off passwords with enough entropy to busy an attacker until the heat death of the universe. OTPs are not meaningfully increasing security in that scenario.
    - nerdawson 4y ago
      That statement directly contradicts the article. A number of accounts would have been accessed were it not for 2FA. The best password won’t save you if a reset email is intercepted. Something you have plus something you know should be what we cone to expect from any service that deals with anything of importance.
      - diarrhea 4y ago
        That doesn’t make too much sense. An intercepted password reset email requires access to the email account. That’s a whole different scenario. If both the email and DO account are behind strong, unique passwords, that won’t happen from a mere DO email leak. Now, if the same, weak passwords are used for both accounts, sure. That’s terrible practice. But it’s also not what the GP was talking about.
        4y ago
helloworld11 4y ago
To anyone who reads this, works in Mailchimp and happens to be in a leadership position at the company, sincerely, it would be nice to see you go bankrupt and fuck right off the face of the landscape. You don't even have the sheer size pretexts of Google, for example, to justify such terrible responsiveness, shitty customer service and generally awful account suspension and service practices. Is it simple laziness or a sharper sort of contempt for your own customers? Disgusting tendency among too many tech companies that reach any moderate size.
- thiscatis 4y ago
  They could have owned the transactional mailing space with Mandrill but Mailchimp'ed it.
- eloff 4y ago
  Having been a victim of this recently, I concur. Don't use MailChimp.
- VWWHFSfQ 4y ago
  > it would be nice to see you go bankrupt I would prefer that they fix their problems instead. They have 800 employees. I don't want those people to lose their jobs.
  - pengaru 4y ago
    > I would prefer that they fix their problems instead. They have 800 employees. I don't want those people to lose their jobs. This thinking is how one rationalizes "bailing out" companies which have objectively failed in the market and should suffer the consequences. Never has it been so easy to form a business and employ people. Gone are the days of a business being some kind of precious thing worth preserving because forming one is so involved. Let the failures experience negative consequences, it's how we improve as a society.
    - VWWHFSfQ 4y ago
      This is a freaking email-sending company. They can fix their problems without firing their whole staff. You're thinking everything is some kind of an existential crisis.
44gg44gg 4y ago
What is a good SMTP service these days in 2022? It beyond frustrating that every provider is now blocking SMTP.
- kitotik 4y ago
  Horribly frustrating. It seems SES or mailgun are the primary options these days.
- podman 4y ago
  I'm pretty happy with Twilio SendGrid.
  - JohnTHaller 4y ago
    Twilio just had 125 customers get hacked thanks to a social engineering attack including ones like Signal.
  - mekster 4y ago
    Happy for a service which only lets you look at 3 days worth of logs until you pay more just to see more logs? Besides, their technical skill is pretty poor when their site shows "page not found" of some sort on log in process for a split second and when you try to search through the logs, they will quickly show you that I've made excessive access after less than 10 searches. They had an incident on themselves and I asked them to resend the emails that they failed to send and support couldn't do that and that got us off of SendGrid. Large free plan limit is the only good part about SendGrid.
    - tpxl 4y ago
      > their technical skill is pretty poor when their site shows "page not found" of some sort on log in process for a split second and when you try to search through the logs This is typical of single page apps. They have a default state of "no data" and then they update it when they get a response.
- yawnxyz 4y ago
yawnxyz 4y ago
I didn't even know Mailchimp supported transactional emails. I thought it was for newsletters and stuff. It's kind of funny that Mailchimp treats a company as large as Digital Ocean as if they're a one person newsletter.
- kawsper 4y ago
  They had another product called Mandrill that did transactional email, they discontinued it, but also allowed old customers (which also paid big $$$) to keep using Mandrill, perhaps that is what DO used? We left Mandrill because they had a DB failure which took them a long time to recover, and we felt that all their focus were on newsletters, and that transactional email didn't get any attention.
  - nerdawson 4y ago
    You’re right, they mention Mandrill in the article. I didn’t realise they’d discontinued it. I remember there being an uproar some years back when they ditched the free tier and made entry level pricing $20/mo.
    - prawn 4y ago
      Or was that Mailgun? We'd used and got frustrated with each, so could be both!
      - nerdawson 4y ago
        Mailgun was the service a number of my clients flocked to when Mandrill went paid-only. Free options seem to be drying up.
        heartbreak 4y ago
        Since then Mailgun also went paid-only, but it’s still practically free. They don’t send an invoice unless it’s over a certain amount, and mine never are.
nulbyte 4y ago
> ... had successfully changed the password, but in the case below, failed to access the account due to the second-factor authentication... Why wasn't two factor authentication required to reset the password? This is Security 101: Greater risks need greater authentication.
smashah 4y ago
As an indie developer, these increasingly frequent security disclosures (although yes good) are getting very frustrating. I want to say this is due to the threat landscape expanding by the day but some part of me suspects that when a service provider becomes 'comfortable' (mailchimp, Heroku, Twilio, etc.) they becomes complacent/cut costs in the security department. The other day I got a clear phishing SMS from REVOLUT! Crazy!
bradgranath 4y ago
Very frustrating that DO was using MailChimp in the first place. Also some nonsense in there about Crypto scammers?
graton 4y ago
I do wish DigitalOcean would support WebAuthn/FIDO2. Meaning I could use my Yubikey and other hardware tokens I have. Instead they only supported TOTP (Google Authenticator is one implementation) second factor which is vulnerable to phishing attacks. But still better than SMS or nothing at all.
- justusw 4y ago
  On a similar note, Azure only supports U2F with yubikey on select Windows / macOS environments, but Firefox ESR on Debian is not supported at all. Every other service I use supports U2F just fine on Linux, but Microsoft wants Linux users to live slightly more insecure.
  - Godel_unicode 4y ago
    https://bugzilla.mozilla.org/show_bug.cgi?id=1530370 https://bugzilla.mozilla.org/show_bug.cgi?id=1530370 Afaict, the bug is that Firefox doesn’t support FIDO2. AzureAD also doesn’t support U2F, which is unsurprising.
shafyy 4y ago
> We have migrated our email services to another provider and are completing thorough security reviews to confirm our vendors’ security posture. Must suck for Mailchimp to lose a big account, but I guess that's not suprising. Mailchimp is going down by a thousand cuts - they could have stayed a great company if they wouldn't have focused on growth so much (I mean they now offer online ship builder and appointment scheduler products).
shantnutiwari 4y ago
Ah Mailchimp. The king of terrible customer service. I was their paid client some years ago, never have I treated as badly (though Convertkit came a close 2nd). People keep recommending Mailchimp, when they are one of the worst companies for support. And funny to see big million dollar corps are treated the same way us plebs are-- at least Mailchimp dont discriminate!

168 points by smitop 4y ago

You would think Digital Ocean being a cloud hosting provider they could do email themselves, secure customer data and not provide it to third parties. Great write up though.

clintonb 4y ago
I would not think that because cloud hosting and email delivery are very different problems to solve.

shawncampbell 4y ago

> on August 8th, our Security Operations team was made aware of a customer who claimed their password had been reset, without their initiation. > One of the first discoveries was a non-DigitalOcean email address that appeared on a regular email from Mailchimp on August 7th. > Soon after we discovered an issue with our Mailchimp account on August 8th, we initiated contact with Mailchimp, both via traditional support channels and other escalation methods. On August 10th, we had our first actionable response

ben_jones 4y ago
> both via traditional support channels and other escalation methods I wish our industry could just be honest that many meaningful escalations have to happen via discriminatory back-channeling, contacting friends, family, former co-workers, ANYONE who might have an in with the organization. Its discriminatory because if you don't know someone you can be SOL.
- xtracto 4y ago
  Mailchimp/Mandrill have specially bad customer support and business user experience. At one point they closed the account of a company I was working at without any previous notice and without any recourse. They just sent a blanket email "Your account has been suspended". They left us scrambling for an alternative (we had both transactional and marketing email). We happily migrated to SES for transactional email and SendGrid for marketing. I no longer recommend or use Mailchimp.
  - 44gg44gg 4y ago
    My issue with SES is that your account may be placed in a Sandbox with no explanation other than to delete your account.
    - Godel_unicode 4y ago
      This is because of fraud. Everyone either does this, or enables the fraudsters. The only third option is to get rid of email, except the fraudsters would just move on to the next thing. Assume that you’ll have availability problems with email and engineer with that in mind.
      - capableweb 4y ago
        Claiming that you either need to ban users without recurse or suffer from enabling fraudsters is why people start feeling like corporations don't have any human employees anymore. If you care about your customers, you could absolutely reach out and verify if the user you are about to/have blocked is a fraudster or not, and then act accordingly. The reason most just say "You're banned, bye" is because they don't want to do that work, and subsequently don't care about their users one bit.
        Godel_unicode 4y ago
        They ban thousands of accounts per day. Every day. If you can solve this problem you will have discovered a license to print money, so by all means try. But the problem is much harder than you are making it sound.
        capableweb 4y ago
        On the other hand, how many do you think would reach out when they get blocked? Most actual fraudsters wouldn't. Of those that do, you can take a look at what they are sending and quickly see if they are actually doing fraud or not. I could say the same thing to you, you're making it sound harder than what it is. But if you're set to the mindset of saving money, I understand minimising work makes sense.
- preommr 4y ago
  > Its discriminatory because if you don't know someone you can be SOL. Thanks for explicitly explaining that part. Now that you've put it that way, it's a pretty apt term but I probably wouldn't have been able to figure out because that term is usually used for things like gender, race, etc. and my brain immediately jumped to that. Maybe I am just stupid, but sometimes it shouldn't be that hard to understand what people mean and it helps to be more explicit.
  - eastbound 4y ago
    Good opportunity to talk about it. An awful lot of people are extremely solitary. I’d wager this is more of a problem than racial discrimination because: - Lonely people aren’t visible, by definition, - Solitude increases racism, so it would be worth solving, - They often end up creating companies and being one’s boss.

xtracto 4y ago

Aaah, so THAT's why their email verification was not working for several hours [1] [2]! [1] https://news.ycombinator.com/item?id=32398773 https://news.ycombinator.com/item?id=32398773 [2] https://news.ycombinator.com/item?id=32392935 https://news.ycombinator.com/item?id=32392935

nerdawson 4y ago

It took two days for a company the size of DO to get an actionable response. What hope do the rest of us have? Interesting write up and 2FA by default sounds like a sensible move. If you’re the type of user to have a DO account, you should be perfectly capable of using 2FA.

throwntoday 4y ago
MFA is inconvenient
- deleted 4y ago
  [deleted]
- teeray 4y ago
  Especially when you generate one-off passwords with enough entropy to busy an attacker until the heat death of the universe. OTPs are not meaningfully increasing security in that scenario.
  - nerdawson 4y ago
    That statement directly contradicts the article. A number of accounts would have been accessed were it not for 2FA. The best password won’t save you if a reset email is intercepted. Something you have plus something you know should be what we cone to expect from any service that deals with anything of importance.
    - diarrhea 4y ago
      That doesn’t make too much sense. An intercepted password reset email requires access to the email account. That’s a whole different scenario. If both the email and DO account are behind strong, unique passwords, that won’t happen from a mere DO email leak. Now, if the same, weak passwords are used for both accounts, sure. That’s terrible practice. But it’s also not what the GP was talking about.
      - 4y ago

helloworld11 4y ago

To anyone who reads this, works in Mailchimp and happens to be in a leadership position at the company, sincerely, it would be nice to see you go bankrupt and fuck right off the face of the landscape. You don't even have the sheer size pretexts of Google, for example, to justify such terrible responsiveness, shitty customer service and generally awful account suspension and service practices. Is it simple laziness or a sharper sort of contempt for your own customers? Disgusting tendency among too many tech companies that reach any moderate size.

thiscatis 4y ago
They could have owned the transactional mailing space with Mandrill but Mailchimp'ed it.
eloff 4y ago
Having been a victim of this recently, I concur. Don't use MailChimp.
VWWHFSfQ 4y ago
> it would be nice to see you go bankrupt I would prefer that they fix their problems instead. They have 800 employees. I don't want those people to lose their jobs.
- pengaru 4y ago
  > I would prefer that they fix their problems instead. They have 800 employees. I don't want those people to lose their jobs. This thinking is how one rationalizes "bailing out" companies which have objectively failed in the market and should suffer the consequences. Never has it been so easy to form a business and employ people. Gone are the days of a business being some kind of precious thing worth preserving because forming one is so involved. Let the failures experience negative consequences, it's how we improve as a society.
  - VWWHFSfQ 4y ago
    This is a freaking email-sending company. They can fix their problems without firing their whole staff. You're thinking everything is some kind of an existential crisis.

44gg44gg 4y ago

What is a good SMTP service these days in 2022? It beyond frustrating that every provider is now blocking SMTP.

kitotik 4y ago
Horribly frustrating. It seems SES or mailgun are the primary options these days.
podman 4y ago
I'm pretty happy with Twilio SendGrid.
- JohnTHaller 4y ago
  Twilio just had 125 customers get hacked thanks to a social engineering attack including ones like Signal.
- mekster 4y ago
  Happy for a service which only lets you look at 3 days worth of logs until you pay more just to see more logs? Besides, their technical skill is pretty poor when their site shows "page not found" of some sort on log in process for a split second and when you try to search through the logs, they will quickly show you that I've made excessive access after less than 10 searches. They had an incident on themselves and I asked them to resend the emails that they failed to send and support couldn't do that and that got us off of SendGrid. Large free plan limit is the only good part about SendGrid.
  - tpxl 4y ago
    > their technical skill is pretty poor when their site shows "page not found" of some sort on log in process for a split second and when you try to search through the logs This is typical of single page apps. They have a default state of "no data" and then they update it when they get a response.
yawnxyz 4y ago

yawnxyz 4y ago

I didn't even know Mailchimp supported transactional emails. I thought it was for newsletters and stuff. It's kind of funny that Mailchimp treats a company as large as Digital Ocean as if they're a one person newsletter.

kawsper 4y ago
They had another product called Mandrill that did transactional email, they discontinued it, but also allowed old customers (which also paid big $$$) to keep using Mandrill, perhaps that is what DO used? We left Mandrill because they had a DB failure which took them a long time to recover, and we felt that all their focus were on newsletters, and that transactional email didn't get any attention.
- nerdawson 4y ago
  You’re right, they mention Mandrill in the article. I didn’t realise they’d discontinued it. I remember there being an uproar some years back when they ditched the free tier and made entry level pricing $20/mo.
  - prawn 4y ago
    Or was that Mailgun? We'd used and got frustrated with each, so could be both!
    - nerdawson 4y ago
      Mailgun was the service a number of my clients flocked to when Mandrill went paid-only. Free options seem to be drying up.
      - heartbreak 4y ago
        Since then Mailgun also went paid-only, but it’s still practically free. They don’t send an invoice unless it’s over a certain amount, and mine never are.

nulbyte 4y ago

> ... had successfully changed the password, but in the case below, failed to access the account due to the second-factor authentication... Why wasn't two factor authentication required to reset the password? This is Security 101: Greater risks need greater authentication.

smashah 4y ago

As an indie developer, these increasingly frequent security disclosures (although yes good) are getting very frustrating. I want to say this is due to the threat landscape expanding by the day but some part of me suspects that when a service provider becomes 'comfortable' (mailchimp, Heroku, Twilio, etc.) they becomes complacent/cut costs in the security department. The other day I got a clear phishing SMS from REVOLUT! Crazy!

bradgranath 4y ago

Very frustrating that DO was using MailChimp in the first place. Also some nonsense in there about Crypto scammers?

graton 4y ago

I do wish DigitalOcean would support WebAuthn/FIDO2. Meaning I could use my Yubikey and other hardware tokens I have. Instead they only supported TOTP (Google Authenticator is one implementation) second factor which is vulnerable to phishing attacks. But still better than SMS or nothing at all.

justusw 4y ago
On a similar note, Azure only supports U2F with yubikey on select Windows / macOS environments, but Firefox ESR on Debian is not supported at all. Every other service I use supports U2F just fine on Linux, but Microsoft wants Linux users to live slightly more insecure.
- Godel_unicode 4y ago
  https://bugzilla.mozilla.org/show_bug.cgi?id=1530370 https://bugzilla.mozilla.org/show_bug.cgi?id=1530370 Afaict, the bug is that Firefox doesn’t support FIDO2. AzureAD also doesn’t support U2F, which is unsurprising.

shafyy 4y ago

> We have migrated our email services to another provider and are completing thorough security reviews to confirm our vendors’ security posture. Must suck for Mailchimp to lose a big account, but I guess that's not suprising. Mailchimp is going down by a thousand cuts - they could have stayed a great company if they wouldn't have focused on growth so much (I mean they now offer online ship builder and appointment scheduler products).

shantnutiwari 4y ago

Ah Mailchimp. The king of terrible customer service. I was their paid client some years ago, never have I treated as badly (though Convertkit came a close 2nd). People keep recommending Mailchimp, when they are one of the worst companies for support. And funny to see big million dollar corps are treated the same way us plebs are-- at least Mailchimp dont discriminate!