5 ms·

One of my favourite days at work in a large bank was when someone sent a boring email to the wrong list. It was a big list. A few people replied all saying "can

One of my favourite days at work in a large bank was when someone sent a boring email to the wrong list. It was a big list. A few people replied all saying "can you remove me from this list please". Then more people replied saying "can you please stop replying all on this thread". Then people said "can you remove me from this list please" again replying all. The mail servers slowed to a crawl. We had thousands of emails. It began at 9am GMT. Eventually after about 3 or so hours everything calmed down and we could send mail again, albeit with a large delay. Around 12:30 the US workers began coming in, openend outlook and saw all the emails. And instead of scrolling down they sent an email, reply all "could you remove me from this list". Again people replied "stop replying all" on reply all. We were up and running again! Mail was down for the rest of the day. The next day some people who were off the day before came in and there was again a brief storm, but sadly this one only lasted around 30 minutes. One of the biggest banks. Lasted a whole day. We christened it "moron storm".

Symbiote 10y ago
Shouldn't these be straightforward to block, by matching on the References header? That contains the message id of previous messages.
- branchless 10y ago
  These are people who in general not tech workers. Also at some points one could not open the email client nor scroll to the latest messages.
brazzledazzle 10y ago
Since these are so trivial to halt and blow away everything in the queue from the thread I can only imagine the sysadmins in charge of these mail systems are letting it continue for comedic effect. That or they have to jump through weeks of change control hoops so letting it play out is faster.
- branchless 10y ago
  No I assume that given the input rate and perhaps due to distributed, non-responsive servers or perhaps sheer incompetence they couldn't halt it.
  - brazzledazzle 10y ago
    It's probably Exchange so nuking the offending messages in the mail queues on the transports during a mail storm is par for the course. Even if they're distributed. The messages already hitting the mailbox hosts would definitely be causing a bad time though.
- pja 10y ago
  Banks have legal constraints on doing that (the need to record all employee communications) which might have made life difficult in this particular case. I agree in general that the mail sysadmins ideally ought to be able to nip this kind of thing in the bud.
  - brazzledazzle 10y ago
    Ah, yes–it's a bank. That's a great point. Not sure what I was thinking. I imagine you could set up a separate transport with special rules that dumps them somewhere outside of the normal mail infrastructure for that message thread but it wouldn't be worth it when the problem happens infrequently and just adding sender restrictions to the list is quick and effective.
desas 10y ago
Morgan Stanley?

Symbiote 10y ago

Shouldn't these be straightforward to block, by matching on the References header? That contains the message id of previous messages.

branchless 10y ago
These are people who in general not tech workers. Also at some points one could not open the email client nor scroll to the latest messages.

brazzledazzle 10y ago

Since these are so trivial to halt and blow away everything in the queue from the thread I can only imagine the sysadmins in charge of these mail systems are letting it continue for comedic effect. That or they have to jump through weeks of change control hoops so letting it play out is faster.

branchless 10y ago
No I assume that given the input rate and perhaps due to distributed, non-responsive servers or perhaps sheer incompetence they couldn't halt it.
- brazzledazzle 10y ago
  It's probably Exchange so nuking the offending messages in the mail queues on the transports during a mail storm is par for the course. Even if they're distributed. The messages already hitting the mailbox hosts would definitely be causing a bad time though.
pja 10y ago
Banks have legal constraints on doing that (the need to record all employee communications) which might have made life difficult in this particular case. I agree in general that the mail sysadmins ideally ought to be able to nip this kind of thing in the bud.
- brazzledazzle 10y ago
  Ah, yes–it's a bank. That's a great point. Not sure what I was thinking. I imagine you could set up a separate transport with special rules that dumps them somewhere outside of the normal mail infrastructure for that message thread but it wouldn't be worth it when the problem happens infrequently and just adding sender restrictions to the list is quick and effective.

desas 10y ago

Morgan Stanley?