11 ms·

Email Storm

69 points by polm23 9y ago

deleted 9y ago
[deleted]
_nalply 9y ago
I saw an email storm on a mailing list for lawyers. Due to a misconfiguration out of office notices weren't suppressed and forwarded to the mailing list. Soon several lawyers who were taking off were contributing to the storm. I immediately saw the self-reinforcing avalanche and told the operator to shut down the mailing list server.
stygiansonic 9y ago
I think almost everyone at a big company has experienced at least one of these. Big email lists should typically have only a set of approved senders.
- protomyth 9y ago
  Its actually fairly easy to setup a decent mailing list without having to have approved senders, but Exchange and its built in functionality is probably the root cause of a lot of these. The ones I have setup are pretty easy to stop since we do some basic checking of the e-mails sent anyway (e.g. attachment too large).
- miheermunjal 9y ago
  O365 (commonly used at large enterprises) makes this dead simple too. You can even have it be self-service, where individuals can empower themselves to have access to a list-serv (its not just "CEOs").
starbuxman 9y ago
This is similar to a group text in iOS/iPhone. It drives me crazy!
- deleted 9y ago
  [deleted]
- ihuman 9y ago
  If everyone is using iMessage, you can leave the group [0]. Unfortunately, you can't leave if someone is using SMS. [0] http://osxdaily.com/2014/09/23/leave-group-message-chat-ios/ http://osxdaily.com/2014/09/23/leave-group-message-chat-ios/
  - protomyth 9y ago
    Yep, its also a pain when you are waiting for a legitimate needed text and cannot leave the group. I guess you can mute a group but I think its named something funny like "Hide Alerts". It is a tad bit beyond the average user because of a poor UI. A friend of mine was in a business meeting on the same day UND announced its new logo and his phone went crazy. The person he was meeting with asked if he should answer it and he said "no, its a lot of bitter, bitter people and UND has no taste.". He did get the contract though.
  - starbuxman 9y ago
    Yeah, but usually there's at least one that's not using iMessage, which makes it a pain ;-)
nancyp 9y ago
Watch out for those out of office notifications replying to group messages and causing flood of out of office replies.
chatmasta 9y ago
We had a few of these in college. My favorite was when someone replied "you know, you can just press M in gmail to mute the conversation." What followed were hundreds of messages saying just, "M". College kids really are the greatest trolls.
Powerofmene 9y ago
When I was with the government if you were on workgroups or had to attend meetings, you could count on 30-40 reply all emails when just trying to schedule meetings or coordinate calendars. For me I received easily 175-225 emails on any given week that should not have included me because of people replying to everyone rather than the one person who needed their info. It drove me crazy and was such a time waster.
ilkkao 9y ago
Saw these storms couple times at Nokia years ago. By far the funniest replies were those that explained correctly what's going on and then concluded that their message should be the end of the thread. What they didn't realize was that the email server was overloaded and that their message would be delivered several hours later together with dozen similar messages from the other people. The next round of people then complained why these people are not stopping.
deleted 9y ago
[deleted]
angelofthe0dd 9y ago
I was hoping to see something in the Wiki article about "Amazon Wallet".
- lorrin 9y ago
  I had the same thought! Anyone recall the details well enough to add an entry? IIRC, it knocked out the Exchange servers entirely for a while.
  - angelofthe0dd 9y ago
    I met the developer who was responsible when I was employed by Amazon. He was trying to reply to an email, but I believe it was a "reply all" (if I recall the details of the story correctly) and the list was massive. Meanwhile, people on that same massive list were also doing a "reply all" to submit their feedback or reply that the email didn't apply to them. Massive email storm resulted. He didn't give all the details but he was ear-to-ear grinning as he told us the story.
asah 9y ago
Edited the wikipedia page to mention Gmail mute, which solves this in practice.
_Codemonkeyism 9y ago
Remember one in a company with 35k employees.
tomphoolery 9y ago
Those examples are definitely cringe-worthy. I wouldn't want to be any of those people!
mantalk 9y ago
"The resulting storm of 'unsubscribe', 'me-too' requests, sarcastic facepalm images and recipes for broccoli casserole resulted in (by the time the list was closed) over 4 million emails and generating over 375GB of network traffic."
DKnoll 9y ago
Why couldn't the sysadmins just kill it with a Transport Rule (silently delete all messages containing the original subject) or temporarily reject messages to the DL? Takes a couple mins.
- eXpl0it3r 9y ago
  That's most likely what happens after the sysadmins get notified of the issue. Take the NHS incident as an example. Within 1h and 15min the generated emails were around 500 million. A one hour response time is still relatively quick. The more important question is, why email systems don't have default rules to prevent such an email storm? When emails go to tens of thousands of people, a system should probably only allow it with special credentials or at least ensure that it's a read-only mailing list. Other measurements would be to detect growing email chains and block emails when the chain has reached some threshold.
  - DKnoll 9y ago
    True enough, but it is fairly easy to restrict use of distribution groups in Exchange, which I would bet most of these orgs use. I'd argue 1hr response is pretty slow for the NHS, since they require 24/7 support availability of even their vendors. I imagine they responded pretty quickly but nobody had a plan in place for situations like this and a large public sector organisation isn't a place conducive to one person taking charge and fixing the problem immediately.
porjo 9y ago
Wouldn't a simple mitigation be to limit the number of recipients allowed in the 'to' and 'cc' fields? 'Bcc' can be used for the mega mailouts. That way, a user hitting reply-all sends to a much more limited set of addresses.

69 points by polm23 9y ago

deleted 9y ago

[deleted]

I saw an email storm on a mailing list for lawyers. Due to a misconfiguration out of office notices weren't suppressed and forwarded to the mailing list. Soon several lawyers who were taking off were contributing to the storm. I immediately saw the self-reinforcing avalanche and told the operator to shut down the mailing list server.

stygiansonic 9y ago

I think almost everyone at a big company has experienced at least one of these. Big email lists should typically have only a set of approved senders.

protomyth 9y ago
Its actually fairly easy to setup a decent mailing list without having to have approved senders, but Exchange and its built in functionality is probably the root cause of a lot of these. The ones I have setup are pretty easy to stop since we do some basic checking of the e-mails sent anyway (e.g. attachment too large).
miheermunjal 9y ago
O365 (commonly used at large enterprises) makes this dead simple too. You can even have it be self-service, where individuals can empower themselves to have access to a list-serv (its not just "CEOs").

starbuxman 9y ago

This is similar to a group text in iOS/iPhone. It drives me crazy!

deleted 9y ago
[deleted]
ihuman 9y ago
If everyone is using iMessage, you can leave the group [0]. Unfortunately, you can't leave if someone is using SMS. [0] http://osxdaily.com/2014/09/23/leave-group-message-chat-ios/ http://osxdaily.com/2014/09/23/leave-group-message-chat-ios/
- protomyth 9y ago
  Yep, its also a pain when you are waiting for a legitimate needed text and cannot leave the group. I guess you can mute a group but I think its named something funny like "Hide Alerts". It is a tad bit beyond the average user because of a poor UI. A friend of mine was in a business meeting on the same day UND announced its new logo and his phone went crazy. The person he was meeting with asked if he should answer it and he said "no, its a lot of bitter, bitter people and UND has no taste.". He did get the contract though.
- starbuxman 9y ago
  Yeah, but usually there's at least one that's not using iMessage, which makes it a pain ;-)

nancyp 9y ago

Watch out for those out of office notifications replying to group messages and causing flood of out of office replies.

chatmasta 9y ago

We had a few of these in college. My favorite was when someone replied "you know, you can just press M in gmail to mute the conversation." What followed were hundreds of messages saying just, "M". College kids really are the greatest trolls.

Powerofmene 9y ago

When I was with the government if you were on workgroups or had to attend meetings, you could count on 30-40 reply all emails when just trying to schedule meetings or coordinate calendars. For me I received easily 175-225 emails on any given week that should not have included me because of people replying to everyone rather than the one person who needed their info. It drove me crazy and was such a time waster.

ilkkao 9y ago

Saw these storms couple times at Nokia years ago. By far the funniest replies were those that explained correctly what's going on and then concluded that their message should be the end of the thread. What they didn't realize was that the email server was overloaded and that their message would be delivered several hours later together with dozen similar messages from the other people. The next round of people then complained why these people are not stopping.

deleted 9y ago

[deleted]

angelofthe0dd 9y ago

I was hoping to see something in the Wiki article about "Amazon Wallet".

lorrin 9y ago
I had the same thought! Anyone recall the details well enough to add an entry? IIRC, it knocked out the Exchange servers entirely for a while.
- angelofthe0dd 9y ago
  I met the developer who was responsible when I was employed by Amazon. He was trying to reply to an email, but I believe it was a "reply all" (if I recall the details of the story correctly) and the list was massive. Meanwhile, people on that same massive list were also doing a "reply all" to submit their feedback or reply that the email didn't apply to them. Massive email storm resulted. He didn't give all the details but he was ear-to-ear grinning as he told us the story.

asah 9y ago

Edited the wikipedia page to mention Gmail mute, which solves this in practice.

_Codemonkeyism 9y ago

Remember one in a company with 35k employees.

tomphoolery 9y ago

Those examples are definitely cringe-worthy. I wouldn't want to be any of those people!

mantalk 9y ago

"The resulting storm of 'unsubscribe', 'me-too' requests, sarcastic facepalm images and recipes for broccoli casserole resulted in (by the time the list was closed) over 4 million emails and generating over 375GB of network traffic."

DKnoll 9y ago

Why couldn't the sysadmins just kill it with a Transport Rule (silently delete all messages containing the original subject) or temporarily reject messages to the DL? Takes a couple mins.

eXpl0it3r 9y ago
That's most likely what happens after the sysadmins get notified of the issue. Take the NHS incident as an example. Within 1h and 15min the generated emails were around 500 million. A one hour response time is still relatively quick. The more important question is, why email systems don't have default rules to prevent such an email storm? When emails go to tens of thousands of people, a system should probably only allow it with special credentials or at least ensure that it's a read-only mailing list. Other measurements would be to detect growing email chains and block emails when the chain has reached some threshold.
- DKnoll 9y ago
  True enough, but it is fairly easy to restrict use of distribution groups in Exchange, which I would bet most of these orgs use. I'd argue 1hr response is pretty slow for the NHS, since they require 24/7 support availability of even their vendors. I imagine they responded pretty quickly but nobody had a plan in place for situations like this and a large public sector organisation isn't a place conducive to one person taking charge and fixing the problem immediately.

porjo 9y ago

Wouldn't a simple mitigation be to limit the number of recipients allowed in the 'to' and 'cc' fields? 'Bcc' can be used for the mega mailouts. That way, a user hitting reply-all sends to a much more limited set of addresses.