19 ms·

The NHS's 1.2M employees are trapped in a 'reply-all' email hell

159 points by agjmills 10y ago

russnewcomer 10y ago
This reminds me of the story Larry Osterman told about Bedlam DL3 [0] at Microsoft. I feel like this is a problem with few good, problem-free technical solutions. [0] https://blogs.technet.microsoft.com/exchange/2004/04/08/me-too/ https://blogs.technet.microsoft.com/exchange/2004/04/08/me-t...
- johansch 10y ago
  "Do you really want to send this e-mail to 1,000,000 people? [Yes, really] [No]" With the latter option being focused by default and emphasized? Yes, doing that is tricky deployment-wise with open protocols, but isn't the whole point of solutions like Exchange to enable system-wide shortcuts like that? With an open solution the next best option is probably to have the mail server send a confirmation request email back to the user with a link to click if they really wanted to send it, instructions on how to reply with a confirmation etc.
  - CydeWeys 10y ago
    It shouldn't even be an option for the vast majority of employees. Rather than their email being a binary permission (you either can or cannot send to members of the domain), it should have more access levels, including a higher access level "can send to MANY members of the domain" that would not be granted to most employees. Most employees would be in the middle tier, which would allow emails sent only up to some number of manageable recipients. And of course the big distribution lists that go to large numbers of employees should be configured so that only a small number of accounts can post to them.
    - drcross 10y ago
      >it should have more access levels. Something about that sounds so fascist.
  - ChuckMcM 10y ago
    At Google they had modified gmail at one point to take an email that was going to go to a lot of people and send it to your manager for approval. That worked pretty well. I believe the data indicated that sending it to anyone else for them to sign off on it helped. You could override a disapproval and send it anyway but that essentially created a willful act that later could be used as part of the reasoning behind your dismissal :-)
    - Too 10y ago
      How many recipients were required to meet the "a lot" threshold? Really depressing place to work at if your manager is spying and micro managing you at this level.
      - koenigdavidmj 10y ago
        An email of that scale I often run by my manager anyway, simply because he's better than I am at writing things in a politically savvy fashion. This is also better than Outlook's solution, which is simply "you can't send to that mailing list at all".
        mioelnir 10y ago
        Where I work the really big lists are all moderated, with only 3-5 people allowed to post to the list. If you need to post something there, you mail one of the moderators the text you want sent on the list, together with an explanation what it is about and why you can't target a smaller number of recipients. Email got a lot more boring since this was introduced.
        johansch 10y ago
        That sounds like the corporate version of the Soviet Union. Oh, you want to have a meeting with more than five people? Please fill out the reason for that on this form and we'll get back to you.
    - wutbrodo 10y ago
  - ejdyksen 10y ago
    This feature exists in Exchange! It's called MailTips. It warns senders in Outlook and Outlook Web App before sending of a number of conditions (including sending to a "large audience"). I was the PM who owned MailTips when it was created/introduced, and our research at the time showed it would be effective for preventing reply-all storms. Implementing this is, of course, harder than you think. The size of groups has to be calculated nightly and distributed to every server, since groups can be nested (and have duplicate recipients).
    - johansch 10y ago
      There you go. :) So this is what it feels like to predict the past... I found that it's enabled and configured to 25 by default. https://technet.microsoft.com/en-us/library/jj649091%28v=exchg.150%29.aspx?f=255&MSPPError=-2147217396 https://technet.microsoft.com/en-us/library/jj649091%28v=exc... (Nice url.) I was assuming the NHS runs on Microsoft. Does anyone know?
      - xapata 10y ago
        It's funny how often it seems like software authors have a time machine. As soon as you get frustrated enough to make a feature request, they go back in time to implement it and pretend like it was always available.
        johansch 10y ago
        From my own experience: Once you reach an audience in the hundreds of millions you're bombarded with edge cases issues daily..you just chose to prioritize addressing them or not. That they make it to the mainstream, which in this case caused us to have this specific conversation is very unusual and can take a lot of time.
      - desas 10y ago
        yes it runs on exchange/ outlook and outlook web access
- brown9-2 10y ago
  Gmail's mute feature is a pretty good countermeasure for this sort of thing, at least at an individual level.
- corecoder 10y ago
  I just had to stop reading, the bad math was too bad (a quarter of 100000 is 13000, then it becomes briefly 25000 and goes back to 13000, and 10% of 13000 is 130...)
  - megablast 10y ago
    There were 55,000 people, but 100,000 mailboxes. 1/4 of 55,000 = 13,0000. That seems to have been where the writer got confused.
dingaling 10y ago
120 replies ( as of lunchtime ) out of 1.2 million recipients is a remarkably low ratio for a non-technology-focused organisation. Looks like there are lot of sensible folk in the NHS, or else they haven't yet all had a chance to check their e-mails between shifts...
- pavel_lishin 10y ago
  I wonder how many emails per day the system usually handles; I would say I probably receive on the order of 100 emails a day (although most of those are automated messages) and probably send half a dozen, and I'm definitely on the low end.
- mikeash 10y ago
  I wonder if most employees lack the needed permission to send an e-mail to 1.2 million recipients. It seems like a smart thing to restrict that kind of action, precisely to avoid havoc. Another possibility is that dealing with messages that have 1.2 million recipients so overloads the system that it it has only been able to deliver 120 of them so far.
- pliu 10y ago
  It's been quite a few years since I was last an Exchange administrator, but I seem to recall that you could secure a distribution list such that only authorized members could send mail to it. Perhaps (hopefully) only a relatively small group at NHS are allowed to mail all 1.2M mailboxes, and within that group there are 120 very silly people.
mfoy_ 10y ago
Something like this seems to happen, and make the news, every couple years... Is it not possible to configure their email server to disallow emails with over X recipients from sending?
- gregmac 10y ago
  It's probably a list, so the actual email has only one recipient. Something essentially expands this list to deliver a separate message to each individual recipient.
  - pavel_lishin 10y ago
    I'm surprised that lists of certain sizes aren't locked down. I would expect the server to refuse to send to "all@nhs.gov.uk" unless the sender is on some sort of pre-approved list.
    - mobiuscog 10y ago
      One would really hope that such an address cannot be externally addressed ;)
      - pavel_lishin 10y ago
        I look forward to another cleverly-named security flaw that allows someone to take down entire e-mail servers with a single externally-sent email.
      - louthy 10y ago
        The domain is nhs.uk ;)
        desus 10y ago
        The domain is .nhs.net ;) Some individual trusts have @trust.nhs.uk addresses, most - probably two thirds use the .nhs.net system.
sjcsjc 10y ago
https://en.m.wikipedia.org/wiki/Email_storm https://en.m.wikipedia.org/wiki/Email_storm
arethuza 10y ago
You can tell they aren't very close to it a they refer to "the" NHS - there are actually four NHSs for different parts of the UK: https://en.wikipedia.org/wiki/National_Health_Service https://en.wikipedia.org/wiki/National_Health_Service
- djmobley 10y ago
  Pretty much everyone in the UK refers to the organization(s) as "the NHS"
  - wlll 10y ago
    Can confirm. It's just "the NHS" wherever you are.
    - DanBC 10y ago
      It's "the NHS" even if you're talking about something not strictly the NHS, such as GP surgeries. And it's "the NHS" when talking about different NHS trusts such as a hospitals trust or a mental health trust.
- bcraven 10y ago
  From your own link: The terms "National Health Service" or "NHS" are also used to refer to the four systems collectively.
  - deleted 10y ago
    [deleted]
  - arethuza 10y ago
    What I meant was which of the four NHSs (which have a lot of different infrastructure) does this apply to? Or to all four - which in itself would be interesting.
    - desas 10y ago
jameshart 10y ago
For context, the NHS is one of the largest employers in the world. The list[1] currently goes: - US Department of Defense - The Chinese Army - Walmart - McDonalds - The NHS And The NHS's staff are all far more likely to actually have to use a computer and email in their day-to-day work than most of those Walmart and McDonalds employees. [1] https://en.wikipedia.org/wiki/List_of_largest_employers https://en.wikipedia.org/wiki/List_of_largest_employers
- tedmiston 10y ago
  As an anecdote, having worked as a contractor for the DoD, the amount of mail received (internally) that is not relevant to your actual work or job is somewhat astronomical, 90–95% for me. There are many, many lists, warnings, notices, digests, alerts, etc.
  - jff 10y ago
    We sometimes get situations where there's a semi-important note in a daily newsletter, and the group secretary decides to copy-paste it in an email to the whole group. Then the level 1 manager doesn't look at the To: line and forwards it to the whole group again. Then our level 2 manager gets the same information from someone else and forwards it to the entire group as well. I'll get the exact same information about today's health and wellness walk or whatever upwards of 3-4 times.
  - Thriptic 10y ago
    Having done some affiliate work with the VA, I can confirm that this is indeed the case.
- bradleyjg 10y ago
  McDonalds doesn't really belong on the list as they include employees of franchises. Or if does than there are some other organizations that belong on the list as well -- such as the Roman Catholic Church.
  - mschuster91 10y ago
econnors 10y ago
I think the iOS "unsubscribe" banner could be adding to the chaos. Not sure about this exact situation, but I've been caught in mailing lists where people instinctually hit the unsubscribe prompt which just sends an email with text saying "unsubscribe", therefor blasting everyone in the thread and not actually unsubscribing them.
- mcintyre1994 10y ago
  It does that as a reply all? Do you have any idea why it does that? EDIT: Maybe it's a mailing list so there's just one address.
- JoshTriplett 10y ago
  That should only happen for lists that have a List-Unsubscribe header. If the iOS mail client does that for arbitrary mails, that seems horribly broken.
stymaar 10y ago
Pro tip to escape this kind of situation : reply to the last email but spoof a fake email address as sender. Now, for everybody who's gonna use the last email to reply (your email then) your email address doesn't appear anywhere. You might need to repeat this process a few times since people don't always reply to your email, but it works pretty well.
- bcraven 10y ago
  >You might need to repeat this process a few times You're suggesting adding millions of extra emails to the system here. I think the best thing to do is wait, or create a rule based on the subject line.
  - stymaar 10y ago
    In this very specific context, your are absolutely right since the main problem is not the little annoyance experimented by the email receivers, but the global overloading of the mailing infrastructure. But this is a really exceptionnel situation, while being trapped in a reply-to-all loop is not that uncommon.
hitgeek 10y ago
I was caught in an email storm like this one time. Someone at a large insurance company accidentally sent an email containing a few patient names/dobs to a group-email mailing list of at least 1000+ outside emails. It took about 2 months before the storm finally ended. It became comical how many responders to the email chastised the original sender for "HIPAA violations" using Reply-All thereby inadvertently committing the same infraction themselves because the original email was included at the bottom of the reply. Also, several times, after 2-3 days of quiet, someone would come back from vacation and reply-all with "I think you sent this to the wrong person", thereby starting the storm all over again. The company of the original sender seemed powerless to stop it, but I always wondered if they could have just disabled/deleted that group-email mailing list.
- SOLAR_FIELDS 10y ago
  Easy to do if there are no permissions or anything else tied to that AD group - but that is often not the case in these kinds of situations.
  - brazzledazzle 10y ago
    Easy enough to setup an authorized senders list. Really any group with a significant number of recipients should have that or failing that some way for a user to remove themselves.
yagyu 10y ago
I just moved to the outlook webmail in a large-ish organization. It surprised me that Reply All is the default reply-button, and a regular reply requires expanding a drop down! The ui really encourages the mistake.
- JoshTriplett 10y ago
  For mails not sent to millions of people, Reply All is the correct default; anyone receiving the original mail should normally see the reply, and narrowing the communication to a subset of people should require thought (and often an explanation). To help avoid this situation, when sending emails to a large number of people (or a list that goes to a large number of people), if you know that responses should not go to that same set of people, use BCC.
  - Too 10y ago
    There is no correct default, show both buttons next to each other with the same size! Now you have to open a tiny drop down to get to normal reply, it's very easy to forget that there is another option there just replying to all without thinking.
- matt_wulfeck 10y ago
  Probably because "reply-all" is almost always the preferred method of response when responding to a group email.
xyzzy4 10y ago
This used to happen all the time when I worked at Cisco... and yes most emails were of the variety "please remove me from this list".
tintor 10y ago
... or (if using gmail) just mute the conversation.
elcct 10y ago
I was caught in 'reply-all' email hell, but not at such scale. It was with about 5000 users. It was hilarious actually, but I can imagine a lot of people will find this annoying.
mkmk 10y ago
Reminds me of this very funny story shared on Metafilter... (if the comment permalink doesn't work, please search for "Reply-All can get even more fun when") http://www.metafilter.com/78177/PLEASE-UNSUBSCRIBE-ME-FROM-THIS-LIST#2408665 http://www.metafilter.com/78177/PLEASE-UNSUBSCRIBE-ME-FROM-T...
smcl 10y ago
The advice at the end is pretty important - if you want the thread to die, just ignore it. All told people mostly followed this, and if only 120 replies were sent in an organisation as huge as the NHS then that's pretty incredible. In our company's "social" list the silly reply-all chains have a very predictable pattern, coming in a series of waves: 1. original sensible email like "hey guys, I have some honey/walnuts/fruit/spirits from my hometown if anyone wants to try/buy" 2. sensible replies start rolling in, but which were accidentally sent to all 3. some jokers/trolls send memes in response (also reply all) 4. the whining begins, people reply-all asking everyone to stop spamming them 5. the memers reply (sometimes with more memes) that it is the optional "social" list that's for this sort of nonserious/fun stuff. 6. some helpful problem-solvers weigh in, sending reply-all instructions (often including MS Paint'd diagrams) on how to unsubscribe from the list or apply an outlook filter 7. finally the "can everyone just stop replying to this though?" emails start, also reply-all (and apparently oblivious to the contradiction/irony of their own reply-all) and everyone participating starts to realise it should be ignored... Once you recognise the pattern it becomes pretty enjoyable identifying which stage of this month's "Emailgate" you're currently at - the whole thing can around to 45-60 mins to play out.
- cgvgffyv 10y ago
  If anyone working on Office 365 OWA is about, please make the Ignore button front and center if the To is a mailing list. One can right click and then Ignore the thread, but presumably nobody is aware of this or we wouldn't be having this conversation.
  - deleted 10y ago
    [deleted]
  - randlet 10y ago
    In my experience watching lay people use computers, they almost never consider right clicking.
    - eru 10y ago
      I guess Apple had a point with the Mac's mouse.
cmgreen 10y ago
I'm surprised they don't have a cap on number of recipients. That is often a problem orgs had to deal with around holiday time.
nneonneo 10y ago
From the BBCs coverage (http://www.bbc.co.uk/news/amp/37979456 http://www.bbc.co.uk/news/amp/37979456), it seems like the problem was caused by a contractor setting up a full-org distribution list by mistake. This DL would have appeared as a single (unfamiliar) email address to most recipients, making it less obvious that "reply-all" would actually send a message to a million receipients. The NHS resolved the issue by killing the offending DL reasonably quickly, which would stop further replies from being generated. This is also the reason why only about a hundred folks (out of a million) replied-all to the message.
- OskarS 10y ago
  Man, i have such sympathy for that nameless engineer whose fault it was. I know exactly the fight-or-flight-like feeling of "oh shit, I fucked up!", even though I've never fucked up on this massive scale.
- nthcolumn 10y ago
  Not the contractors fault at all! People make mistakes. If your design depends on user vigilance then its YOUR fault. People make mistakes! A single 'rogue' should not be able to take down the entire show just by sending an email. The problem was exacerbated mainly the software being used made by Microsoft - a fairly ubiquitous but non-standard, proprietary system. It has no place in a public entity like the NHS. Sack the highly paid idiots who procured it at great expense over cheaper and higher quality alternatives instead.
blauditore 10y ago
I received such a mail from Microsoft a few years back. It was something related to WP/W8 development with thousands or tens of thousands (IIRC) of email addresses in the "to" field, so probably all registered app devs. I tried to send a witty response, but my mail server (ironically on outlook.com) refused to do it.
- jessaustin 10y ago
  Publicly revealing all those addresses in the "To:" could be considered really abusive. That's a screw-up above and beyond the spamming.
rickette 10y ago
I assume all mails run over NHS controlled mailservers. Some administrator could just step up and kill/block the email thread, right? Shouldn't be that hard of a problem...
- desus 10y ago
  That's exactly what they did. "The distribution list has been removed and associated emails are being traced and cleared." See http://support.nhs.net/servicestatus http://support.nhs.net/servicestatus
eggie5 10y ago
This happened at Qualcomm once. I think it was an IT email too. It went out to most of the corp and what really started the issue was people's vacation/out-of-office auto-reply-all messages -- those really started magnifying problem. Then people started making it even worse by reply-all sending "remove me from this list!" etc. The most funny messages were the reply-all that said "Stop sending reply all to the whole list!" It lasted most of the day...
- corecoder 10y ago
  By the way, does anybody know of a valid reason why vacation/out-of-office messages shouldn't be forbidden forever? Has any organization ever had any positive outcome whatsoever from their members having those autoreplies?
golemotron 10y ago
This might be the most clever Denial of Service Attack ever devised.
deleted 10y ago
[deleted]
DonHopkins 10y ago
Back in the days of ARPANET mailing lists, there used to be an "educational" mailing list called "please-remove-me", that was for people who asked an entire mailing list to remove them, instead of removing themselves, or sending email to the administrative "-request" address. So when somebody asked an entire mailing list to remove them, somebody else would add them to the "please-remove-me" mailing list, and they would start getting hundreds of "please remove me" requests from other people, so they could discuss the topic of being removed from mailing lists with people with similar interests, without bothering people on mailing lists whose topics weren't about being removed from mailing lists. It worked so well that it was a victim of its own success: Eventually the "please-remove-me" mailing list was so popular that it got too big and had to be shut down... ...Then there was Jordan Hubbard's infamous "rwall incident" in 1987: http://everything2.com/title/Jordan+K.+Hubbard http://everything2.com/title/Jordan+K.+Hubbard
jplahn 10y ago
Ah. This reminds me of a great story in Amazon of a conference room booking email gone wrong. The email was inadvertently sent to the entire company, so that employees in Japan, China, Scotland, France, etc. all knew of this fabled booking. At its peak over 280k emails were being delivered throughout the company. Also good to remember: replying UNSUBSCRIBE to an email list is typically not how you go about it.
- wutbrodo 10y ago
  I remember this happening at Google, but I'm not sure if it was the entire company or just something like eng-announce. I myself once had the wrong CL # on my clipboard and attached a random CL of mine to a bug that Jeff Dean had opened about a development tool used by every engineer (with very active discussion across the engineer on the bug). I don't remember these incidents being treated with anything but mild amusement, certainly not "email hell". This probably has something to do with the fact that GMail makes it pretty easy to block an email thread.
nkrisc 10y ago
I was once part of a ~20k person reply-all chain. After a while I sent an email to each person who sent yet another reply-all asking to be removed an individual email explaining how dumb they were, in the nicest, most professionally appropriate way possible. None of them replied to me.
iamatworknow 10y ago
When I was in college e-mail was everything. Facebook was only just getting off the ground, so when an event was happening or a club was recruiting the best way to get the word out was to send an e-mail to the listserv, which then sent it to specific class years or to the whole campus. Problem was this listserv wasn't moderated at all. Any club president or anyone else who wanted access could get it, and the messages weren't screened for content before being distributed to every e-mail address on the list. One such club president wrote an e-mail to the ex of his current girl that began, "Me and Jessica fucked like rabbits..." and he accidentally included the listserv address for the entire campus on that e-mail. The guy was generally well known as an active student with great grades who didn't get in trouble, so when a catty lovers quarrel was made public to thousands of students and faculty by his own hand it was kinda entertaining.
cs02rm0 10y ago
My wife works for the NHS so I had a quick look at her inbox (as it were). Thoroughly disappointing. Normally when I see this happen it's at relatively IT-literate organisations. The NHS emails all seem to be single replies to the original email asking to be removed / saying it seems to be sent in error. I say disappointing because I didn't see any threads spiralling off into back-and-forth between people asking others to stop replying or anyone trolling others. Such a waste.
enraged_camel 10y ago
Email clients need a way to remove the "reply-all" button. That way, users are forced to manually enter the emails of people other than the sender, which would make them think about who actually needs to be copied.
Spooky23 10y ago
It's amazing to me that Microsoft never fixed this issue with Exchange. There are a few edge cases like this where users with no ill intent can do lots of damage. Strong procedural controls are really the only defense.
jlebrech 10y ago
surely the exchange server can block reply alls that have too many recipients, or maybe mute people who've already chipped in for 24hrs.
namesbc 10y ago
How about responding to the list with everyone on BCC stating "You have been unsubscribed."
j45 10y ago
It's a little odd that this has received media attention. What I'd do: independently of the email, ask for a procedure to be sent out to create an email filter/rule. Wouldn't take more than 1-2 minutes to complete for each end user.
giarc 10y ago
I work for a large organization and this happens once or twice a year. I find that when speaking with people, no one knows what BCC is for. Many of these would be avoided if people used BCC properly.
whack 10y ago
It's amusing how many people continue to suffer from a solved problem. Any email system that resembles Gmail in its most basic form, won't have any such problems. 100 people replying-all to the same email? Oh hey, they are all lumped together into a single thread which you can easily ignore. Don't want to see any more emails from that thread? Oh hey, you can just hit the mute button on that thread, and you will never see any more emails there. It boggles my mind how many people and organizations continue to use outdated crappy systems when vastly superior alternatives have been publicly available for over a decade.
- lmm 10y ago
  We're talking about a European organization that handles people's most sensitive personal data - shipping that to the US to be scanned by Google and the NSA is not an option.
  - eru 10y ago
    OP didn't talk about using gmail. Only about copying gmail's features.
- dpark 10y ago
  Outlook supports threading and "ignore". This still happens. If you put a few thousand users in Gmail for work (or whatever they call it), it'll happen there, too. Because it's easier to reply all than to think the issue through, remember that ignore is an option, and use it.
- theandrewbailey 10y ago
  Better software would not have helped in this situation. Most employees in non-technical organizations are (guess what?) non-technical. Most want to do duties reflective of their job title, and by and large, view computers and software as tools, and not things to be known or studied.
sandworm101 10y ago
>> 120 replies so far — meaning that more than 140 million needless emails have been sent across the NHS's network today. As a result, they said, its email systems are running "very slow today." Is this 1994? That isn't my definition of email hell. 100 useless emails in my inbox is a bad day, but it certainly isn't a hellish one. And 140 million emails shouldn't slow anything down. That;s a drop in the spam bucket for any organization the size of the NHS.
jessewmc 10y ago
Tangent: I found 1.4M to 1.7M employees for NHS on google. 64.1M people in the UK, 31.42M employed. 1.4M / 31.42M = 4.5% of the entire UK workforce works for NHS? Is this a normal distribution for a developed country?
- martinald 10y ago
  How many people are employed in the US in healthcare? I would guess way more (10%? 20%?) We spend about 7% of GDP on the NHS. So it's about right that number.
- sockmeistr 10y ago
  Approximately 9% of US workers are in healthcare. (http://kff.org/other/state-indicator/health-care-employment-as-total/ http://kff.org/other/state-indicator/health-care-employment-...) The UK does have a private healthcare sector, although smaller than the NHS.
todd8 10y ago
I once went to a crowed midnight movie showing of the 1950's "It Came From Outerspace". The crowd seemed quite intoxicated and so I missed the first 40 minutes of the movie's dialog because every few seconds someone would stand up and yell SHUT UP and then someone else would yell back from the other side of the theater NO YOU SHUT UP. It was terrible and amusing at the same time.
teddyh 10y ago
Ornk! http://www.threepanelsoul.com/comic/on-sea-lions http://www.threepanelsoul.com/comic/on-sea-lions
deleted 10y ago
[deleted]
darkr 10y ago
This is why you should access controls on configure large mail groups. I thought this was pretty standard at large organisations. A select few people can actually send to a given group, but anyone else will just receive a bounceback.
Thriptic 10y ago
gg no re?

159 points by agjmills 10y ago

This reminds me of the story Larry Osterman told about Bedlam DL3 [0] at Microsoft. I feel like this is a problem with few good, problem-free technical solutions. [0] https://blogs.technet.microsoft.com/exchange/2004/04/08/me-too/ https://blogs.technet.microsoft.com/exchange/2004/04/08/me-t...

johansch 10y ago
"Do you really want to send this e-mail to 1,000,000 people? [Yes, really] [No]" With the latter option being focused by default and emphasized? Yes, doing that is tricky deployment-wise with open protocols, but isn't the whole point of solutions like Exchange to enable system-wide shortcuts like that? With an open solution the next best option is probably to have the mail server send a confirmation request email back to the user with a link to click if they really wanted to send it, instructions on how to reply with a confirmation etc.
- CydeWeys 10y ago
  It shouldn't even be an option for the vast majority of employees. Rather than their email being a binary permission (you either can or cannot send to members of the domain), it should have more access levels, including a higher access level "can send to MANY members of the domain" that would not be granted to most employees. Most employees would be in the middle tier, which would allow emails sent only up to some number of manageable recipients. And of course the big distribution lists that go to large numbers of employees should be configured so that only a small number of accounts can post to them.
  - drcross 10y ago
    >it should have more access levels. Something about that sounds so fascist.
- ChuckMcM 10y ago
  At Google they had modified gmail at one point to take an email that was going to go to a lot of people and send it to your manager for approval. That worked pretty well. I believe the data indicated that sending it to anyone else for them to sign off on it helped. You could override a disapproval and send it anyway but that essentially created a willful act that later could be used as part of the reasoning behind your dismissal :-)
  - Too 10y ago
    How many recipients were required to meet the "a lot" threshold? Really depressing place to work at if your manager is spying and micro managing you at this level.
    - koenigdavidmj 10y ago
      An email of that scale I often run by my manager anyway, simply because he's better than I am at writing things in a politically savvy fashion. This is also better than Outlook's solution, which is simply "you can't send to that mailing list at all".
      - mioelnir 10y ago
        Where I work the really big lists are all moderated, with only 3-5 people allowed to post to the list. If you need to post something there, you mail one of the moderators the text you want sent on the list, together with an explanation what it is about and why you can't target a smaller number of recipients. Email got a lot more boring since this was introduced.
        johansch 10y ago
        That sounds like the corporate version of the Soviet Union. Oh, you want to have a meeting with more than five people? Please fill out the reason for that on this form and we'll get back to you.
  - wutbrodo 10y ago
- ejdyksen 10y ago
  This feature exists in Exchange! It's called MailTips. It warns senders in Outlook and Outlook Web App before sending of a number of conditions (including sending to a "large audience"). I was the PM who owned MailTips when it was created/introduced, and our research at the time showed it would be effective for preventing reply-all storms. Implementing this is, of course, harder than you think. The size of groups has to be calculated nightly and distributed to every server, since groups can be nested (and have duplicate recipients).
  - johansch 10y ago
    There you go. :) So this is what it feels like to predict the past... I found that it's enabled and configured to 25 by default. https://technet.microsoft.com/en-us/library/jj649091%28v=exchg.150%29.aspx?f=255&MSPPError=-2147217396 https://technet.microsoft.com/en-us/library/jj649091%28v=exc... (Nice url.) I was assuming the NHS runs on Microsoft. Does anyone know?
    - xapata 10y ago
      It's funny how often it seems like software authors have a time machine. As soon as you get frustrated enough to make a feature request, they go back in time to implement it and pretend like it was always available.
      - johansch 10y ago
        From my own experience: Once you reach an audience in the hundreds of millions you're bombarded with edge cases issues daily..you just chose to prioritize addressing them or not. That they make it to the mainstream, which in this case caused us to have this specific conversation is very unusual and can take a lot of time.
    - desas 10y ago
      yes it runs on exchange/ outlook and outlook web access
brown9-2 10y ago
Gmail's mute feature is a pretty good countermeasure for this sort of thing, at least at an individual level.
corecoder 10y ago
I just had to stop reading, the bad math was too bad (a quarter of 100000 is 13000, then it becomes briefly 25000 and goes back to 13000, and 10% of 13000 is 130...)
- megablast 10y ago
  There were 55,000 people, but 100,000 mailboxes. 1/4 of 55,000 = 13,0000. That seems to have been where the writer got confused.

dingaling 10y ago

120 replies ( as of lunchtime ) out of 1.2 million recipients is a remarkably low ratio for a non-technology-focused organisation. Looks like there are lot of sensible folk in the NHS, or else they haven't yet all had a chance to check their e-mails between shifts...

pavel_lishin 10y ago
I wonder how many emails per day the system usually handles; I would say I probably receive on the order of 100 emails a day (although most of those are automated messages) and probably send half a dozen, and I'm definitely on the low end.
mikeash 10y ago
I wonder if most employees lack the needed permission to send an e-mail to 1.2 million recipients. It seems like a smart thing to restrict that kind of action, precisely to avoid havoc. Another possibility is that dealing with messages that have 1.2 million recipients so overloads the system that it it has only been able to deliver 120 of them so far.
pliu 10y ago
It's been quite a few years since I was last an Exchange administrator, but I seem to recall that you could secure a distribution list such that only authorized members could send mail to it. Perhaps (hopefully) only a relatively small group at NHS are allowed to mail all 1.2M mailboxes, and within that group there are 120 very silly people.

mfoy_ 10y ago

Something like this seems to happen, and make the news, every couple years... Is it not possible to configure their email server to disallow emails with over X recipients from sending?

gregmac 10y ago
It's probably a list, so the actual email has only one recipient. Something essentially expands this list to deliver a separate message to each individual recipient.
- pavel_lishin 10y ago
  I'm surprised that lists of certain sizes aren't locked down. I would expect the server to refuse to send to "all@nhs.gov.uk" unless the sender is on some sort of pre-approved list.
  - mobiuscog 10y ago
    One would really hope that such an address cannot be externally addressed ;)
    - pavel_lishin 10y ago
      I look forward to another cleverly-named security flaw that allows someone to take down entire e-mail servers with a single externally-sent email.
    - louthy 10y ago
      The domain is nhs.uk ;)
      - desus 10y ago
        The domain is .nhs.net ;) Some individual trusts have @trust.nhs.uk addresses, most - probably two thirds use the .nhs.net system.

sjcsjc 10y ago

https://en.m.wikipedia.org/wiki/Email_storm https://en.m.wikipedia.org/wiki/Email_storm

arethuza 10y ago

You can tell they aren't very close to it a they refer to "the" NHS - there are actually four NHSs for different parts of the UK: https://en.wikipedia.org/wiki/National_Health_Service https://en.wikipedia.org/wiki/National_Health_Service

djmobley 10y ago
Pretty much everyone in the UK refers to the organization(s) as "the NHS"
- wlll 10y ago
  Can confirm. It's just "the NHS" wherever you are.
  - DanBC 10y ago
    It's "the NHS" even if you're talking about something not strictly the NHS, such as GP surgeries. And it's "the NHS" when talking about different NHS trusts such as a hospitals trust or a mental health trust.
bcraven 10y ago
From your own link: The terms "National Health Service" or "NHS" are also used to refer to the four systems collectively.
- deleted 10y ago
  [deleted]
- arethuza 10y ago
  What I meant was which of the four NHSs (which have a lot of different infrastructure) does this apply to? Or to all four - which in itself would be interesting.
  - desas 10y ago

jameshart 10y ago

For context, the NHS is one of the largest employers in the world. The list[1] currently goes: - US Department of Defense - The Chinese Army - Walmart - McDonalds - The NHS And The NHS's staff are all far more likely to actually have to use a computer and email in their day-to-day work than most of those Walmart and McDonalds employees. [1] https://en.wikipedia.org/wiki/List_of_largest_employers https://en.wikipedia.org/wiki/List_of_largest_employers

tedmiston 10y ago
As an anecdote, having worked as a contractor for the DoD, the amount of mail received (internally) that is not relevant to your actual work or job is somewhat astronomical, 90–95% for me. There are many, many lists, warnings, notices, digests, alerts, etc.
- jff 10y ago
  We sometimes get situations where there's a semi-important note in a daily newsletter, and the group secretary decides to copy-paste it in an email to the whole group. Then the level 1 manager doesn't look at the To: line and forwards it to the whole group again. Then our level 2 manager gets the same information from someone else and forwards it to the entire group as well. I'll get the exact same information about today's health and wellness walk or whatever upwards of 3-4 times.
- Thriptic 10y ago
  Having done some affiliate work with the VA, I can confirm that this is indeed the case.
bradleyjg 10y ago
McDonalds doesn't really belong on the list as they include employees of franchises. Or if does than there are some other organizations that belong on the list as well -- such as the Roman Catholic Church.
- mschuster91 10y ago

econnors 10y ago

I think the iOS "unsubscribe" banner could be adding to the chaos. Not sure about this exact situation, but I've been caught in mailing lists where people instinctually hit the unsubscribe prompt which just sends an email with text saying "unsubscribe", therefor blasting everyone in the thread and not actually unsubscribing them.

mcintyre1994 10y ago
It does that as a reply all? Do you have any idea why it does that? EDIT: Maybe it's a mailing list so there's just one address.
JoshTriplett 10y ago
That should only happen for lists that have a List-Unsubscribe header. If the iOS mail client does that for arbitrary mails, that seems horribly broken.

stymaar 10y ago

Pro tip to escape this kind of situation : reply to the last email but spoof a fake email address as sender. Now, for everybody who's gonna use the last email to reply (your email then) your email address doesn't appear anywhere. You might need to repeat this process a few times since people don't always reply to your email, but it works pretty well.

bcraven 10y ago
>You might need to repeat this process a few times You're suggesting adding millions of extra emails to the system here. I think the best thing to do is wait, or create a rule based on the subject line.
- stymaar 10y ago
  In this very specific context, your are absolutely right since the main problem is not the little annoyance experimented by the email receivers, but the global overloading of the mailing infrastructure. But this is a really exceptionnel situation, while being trapped in a reply-to-all loop is not that uncommon.

hitgeek 10y ago

I was caught in an email storm like this one time. Someone at a large insurance company accidentally sent an email containing a few patient names/dobs to a group-email mailing list of at least 1000+ outside emails. It took about 2 months before the storm finally ended. It became comical how many responders to the email chastised the original sender for "HIPAA violations" using Reply-All thereby inadvertently committing the same infraction themselves because the original email was included at the bottom of the reply. Also, several times, after 2-3 days of quiet, someone would come back from vacation and reply-all with "I think you sent this to the wrong person", thereby starting the storm all over again. The company of the original sender seemed powerless to stop it, but I always wondered if they could have just disabled/deleted that group-email mailing list.

SOLAR_FIELDS 10y ago
Easy to do if there are no permissions or anything else tied to that AD group - but that is often not the case in these kinds of situations.
- brazzledazzle 10y ago
  Easy enough to setup an authorized senders list. Really any group with a significant number of recipients should have that or failing that some way for a user to remove themselves.

yagyu 10y ago

I just moved to the outlook webmail in a large-ish organization. It surprised me that Reply All is the default reply-button, and a regular reply requires expanding a drop down! The ui really encourages the mistake.

JoshTriplett 10y ago
For mails not sent to millions of people, Reply All is the correct default; anyone receiving the original mail should normally see the reply, and narrowing the communication to a subset of people should require thought (and often an explanation). To help avoid this situation, when sending emails to a large number of people (or a list that goes to a large number of people), if you know that responses should not go to that same set of people, use BCC.
- Too 10y ago
  There is no correct default, show both buttons next to each other with the same size! Now you have to open a tiny drop down to get to normal reply, it's very easy to forget that there is another option there just replying to all without thinking.
matt_wulfeck 10y ago
Probably because "reply-all" is almost always the preferred method of response when responding to a group email.

xyzzy4 10y ago

This used to happen all the time when I worked at Cisco... and yes most emails were of the variety "please remove me from this list".

tintor 10y ago

... or (if using gmail) just mute the conversation.

elcct 10y ago

I was caught in 'reply-all' email hell, but not at such scale. It was with about 5000 users. It was hilarious actually, but I can imagine a lot of people will find this annoying.

mkmk 10y ago

Reminds me of this very funny story shared on Metafilter... (if the comment permalink doesn't work, please search for "Reply-All can get even more fun when") http://www.metafilter.com/78177/PLEASE-UNSUBSCRIBE-ME-FROM-THIS-LIST#2408665 http://www.metafilter.com/78177/PLEASE-UNSUBSCRIBE-ME-FROM-T...

smcl 10y ago

The advice at the end is pretty important - if you want the thread to die, just ignore it. All told people mostly followed this, and if only 120 replies were sent in an organisation as huge as the NHS then that's pretty incredible. In our company's "social" list the silly reply-all chains have a very predictable pattern, coming in a series of waves: 1. original sensible email like "hey guys, I have some honey/walnuts/fruit/spirits from my hometown if anyone wants to try/buy" 2. sensible replies start rolling in, but which were accidentally sent to all 3. some jokers/trolls send memes in response (also reply all) 4. the whining begins, people reply-all asking everyone to stop spamming them 5. the memers reply (sometimes with more memes) that it is the optional "social" list that's for this sort of nonserious/fun stuff. 6. some helpful problem-solvers weigh in, sending reply-all instructions (often including MS Paint'd diagrams) on how to unsubscribe from the list or apply an outlook filter 7. finally the "can everyone just stop replying to this though?" emails start, also reply-all (and apparently oblivious to the contradiction/irony of their own reply-all) and everyone participating starts to realise it should be ignored... Once you recognise the pattern it becomes pretty enjoyable identifying which stage of this month's "Emailgate" you're currently at - the whole thing can around to 45-60 mins to play out.

cgvgffyv 10y ago
If anyone working on Office 365 OWA is about, please make the Ignore button front and center if the To is a mailing list. One can right click and then Ignore the thread, but presumably nobody is aware of this or we wouldn't be having this conversation.
- deleted 10y ago
  [deleted]
- randlet 10y ago
  In my experience watching lay people use computers, they almost never consider right clicking.
  - eru 10y ago
    I guess Apple had a point with the Mac's mouse.

cmgreen 10y ago

I'm surprised they don't have a cap on number of recipients. That is often a problem orgs had to deal with around holiday time.

nneonneo 10y ago

From the BBCs coverage (http://www.bbc.co.uk/news/amp/37979456 http://www.bbc.co.uk/news/amp/37979456), it seems like the problem was caused by a contractor setting up a full-org distribution list by mistake. This DL would have appeared as a single (unfamiliar) email address to most recipients, making it less obvious that "reply-all" would actually send a message to a million receipients. The NHS resolved the issue by killing the offending DL reasonably quickly, which would stop further replies from being generated. This is also the reason why only about a hundred folks (out of a million) replied-all to the message.

OskarS 10y ago
Man, i have such sympathy for that nameless engineer whose fault it was. I know exactly the fight-or-flight-like feeling of "oh shit, I fucked up!", even though I've never fucked up on this massive scale.
nthcolumn 10y ago
Not the contractors fault at all! People make mistakes. If your design depends on user vigilance then its YOUR fault. People make mistakes! A single 'rogue' should not be able to take down the entire show just by sending an email. The problem was exacerbated mainly the software being used made by Microsoft - a fairly ubiquitous but non-standard, proprietary system. It has no place in a public entity like the NHS. Sack the highly paid idiots who procured it at great expense over cheaper and higher quality alternatives instead.

blauditore 10y ago

I received such a mail from Microsoft a few years back. It was something related to WP/W8 development with thousands or tens of thousands (IIRC) of email addresses in the "to" field, so probably all registered app devs. I tried to send a witty response, but my mail server (ironically on outlook.com) refused to do it.

jessaustin 10y ago
Publicly revealing all those addresses in the "To:" could be considered really abusive. That's a screw-up above and beyond the spamming.

rickette 10y ago

I assume all mails run over NHS controlled mailservers. Some administrator could just step up and kill/block the email thread, right? Shouldn't be that hard of a problem...

desus 10y ago
That's exactly what they did. "The distribution list has been removed and associated emails are being traced and cleared." See http://support.nhs.net/servicestatus http://support.nhs.net/servicestatus

eggie5 10y ago

This happened at Qualcomm once. I think it was an IT email too. It went out to most of the corp and what really started the issue was people's vacation/out-of-office auto-reply-all messages -- those really started magnifying problem. Then people started making it even worse by reply-all sending "remove me from this list!" etc. The most funny messages were the reply-all that said "Stop sending reply all to the whole list!" It lasted most of the day...

corecoder 10y ago
By the way, does anybody know of a valid reason why vacation/out-of-office messages shouldn't be forbidden forever? Has any organization ever had any positive outcome whatsoever from their members having those autoreplies?

golemotron 10y ago

This might be the most clever Denial of Service Attack ever devised.

deleted 10y ago

[deleted]

DonHopkins 10y ago

Back in the days of ARPANET mailing lists, there used to be an "educational" mailing list called "please-remove-me", that was for people who asked an entire mailing list to remove them, instead of removing themselves, or sending email to the administrative "-request" address. So when somebody asked an entire mailing list to remove them, somebody else would add them to the "please-remove-me" mailing list, and they would start getting hundreds of "please remove me" requests from other people, so they could discuss the topic of being removed from mailing lists with people with similar interests, without bothering people on mailing lists whose topics weren't about being removed from mailing lists. It worked so well that it was a victim of its own success: Eventually the "please-remove-me" mailing list was so popular that it got too big and had to be shut down... ...Then there was Jordan Hubbard's infamous "rwall incident" in 1987: http://everything2.com/title/Jordan+K.+Hubbard http://everything2.com/title/Jordan+K.+Hubbard

jplahn 10y ago

Ah. This reminds me of a great story in Amazon of a conference room booking email gone wrong. The email was inadvertently sent to the entire company, so that employees in Japan, China, Scotland, France, etc. all knew of this fabled booking. At its peak over 280k emails were being delivered throughout the company. Also good to remember: replying UNSUBSCRIBE to an email list is typically not how you go about it.

wutbrodo 10y ago
I remember this happening at Google, but I'm not sure if it was the entire company or just something like eng-announce. I myself once had the wrong CL # on my clipboard and attached a random CL of mine to a bug that Jeff Dean had opened about a development tool used by every engineer (with very active discussion across the engineer on the bug). I don't remember these incidents being treated with anything but mild amusement, certainly not "email hell". This probably has something to do with the fact that GMail makes it pretty easy to block an email thread.

nkrisc 10y ago

I was once part of a ~20k person reply-all chain. After a while I sent an email to each person who sent yet another reply-all asking to be removed an individual email explaining how dumb they were, in the nicest, most professionally appropriate way possible. None of them replied to me.

iamatworknow 10y ago

When I was in college e-mail was everything. Facebook was only just getting off the ground, so when an event was happening or a club was recruiting the best way to get the word out was to send an e-mail to the listserv, which then sent it to specific class years or to the whole campus. Problem was this listserv wasn't moderated at all. Any club president or anyone else who wanted access could get it, and the messages weren't screened for content before being distributed to every e-mail address on the list. One such club president wrote an e-mail to the ex of his current girl that began, "Me and Jessica fucked like rabbits..." and he accidentally included the listserv address for the entire campus on that e-mail. The guy was generally well known as an active student with great grades who didn't get in trouble, so when a catty lovers quarrel was made public to thousands of students and faculty by his own hand it was kinda entertaining.

cs02rm0 10y ago

My wife works for the NHS so I had a quick look at her inbox (as it were). Thoroughly disappointing. Normally when I see this happen it's at relatively IT-literate organisations. The NHS emails all seem to be single replies to the original email asking to be removed / saying it seems to be sent in error. I say disappointing because I didn't see any threads spiralling off into back-and-forth between people asking others to stop replying or anyone trolling others. Such a waste.

enraged_camel 10y ago

Email clients need a way to remove the "reply-all" button. That way, users are forced to manually enter the emails of people other than the sender, which would make them think about who actually needs to be copied.

Spooky23 10y ago

It's amazing to me that Microsoft never fixed this issue with Exchange. There are a few edge cases like this where users with no ill intent can do lots of damage. Strong procedural controls are really the only defense.

jlebrech 10y ago

surely the exchange server can block reply alls that have too many recipients, or maybe mute people who've already chipped in for 24hrs.

namesbc 10y ago

How about responding to the list with everyone on BCC stating "You have been unsubscribed."

j45 10y ago

It's a little odd that this has received media attention. What I'd do: independently of the email, ask for a procedure to be sent out to create an email filter/rule. Wouldn't take more than 1-2 minutes to complete for each end user.

giarc 10y ago

I work for a large organization and this happens once or twice a year. I find that when speaking with people, no one knows what BCC is for. Many of these would be avoided if people used BCC properly.

whack 10y ago

It's amusing how many people continue to suffer from a solved problem. Any email system that resembles Gmail in its most basic form, won't have any such problems. 100 people replying-all to the same email? Oh hey, they are all lumped together into a single thread which you can easily ignore. Don't want to see any more emails from that thread? Oh hey, you can just hit the mute button on that thread, and you will never see any more emails there. It boggles my mind how many people and organizations continue to use outdated crappy systems when vastly superior alternatives have been publicly available for over a decade.

lmm 10y ago
We're talking about a European organization that handles people's most sensitive personal data - shipping that to the US to be scanned by Google and the NSA is not an option.
- eru 10y ago
  OP didn't talk about using gmail. Only about copying gmail's features.
dpark 10y ago
Outlook supports threading and "ignore". This still happens. If you put a few thousand users in Gmail for work (or whatever they call it), it'll happen there, too. Because it's easier to reply all than to think the issue through, remember that ignore is an option, and use it.
theandrewbailey 10y ago
Better software would not have helped in this situation. Most employees in non-technical organizations are (guess what?) non-technical. Most want to do duties reflective of their job title, and by and large, view computers and software as tools, and not things to be known or studied.

sandworm101 10y ago

>> 120 replies so far — meaning that more than 140 million needless emails have been sent across the NHS's network today. As a result, they said, its email systems are running "very slow today." Is this 1994? That isn't my definition of email hell. 100 useless emails in my inbox is a bad day, but it certainly isn't a hellish one. And 140 million emails shouldn't slow anything down. That;s a drop in the spam bucket for any organization the size of the NHS.

jessewmc 10y ago

Tangent: I found 1.4M to 1.7M employees for NHS on google. 64.1M people in the UK, 31.42M employed. 1.4M / 31.42M = 4.5% of the entire UK workforce works for NHS? Is this a normal distribution for a developed country?

martinald 10y ago
How many people are employed in the US in healthcare? I would guess way more (10%? 20%?) We spend about 7% of GDP on the NHS. So it's about right that number.
sockmeistr 10y ago
Approximately 9% of US workers are in healthcare. (http://kff.org/other/state-indicator/health-care-employment-as-total/ http://kff.org/other/state-indicator/health-care-employment-...) The UK does have a private healthcare sector, although smaller than the NHS.

todd8 10y ago

I once went to a crowed midnight movie showing of the 1950's "It Came From Outerspace". The crowd seemed quite intoxicated and so I missed the first 40 minutes of the movie's dialog because every few seconds someone would stand up and yell SHUT UP and then someone else would yell back from the other side of the theater NO YOU SHUT UP. It was terrible and amusing at the same time.

teddyh 10y ago

Ornk! http://www.threepanelsoul.com/comic/on-sea-lions http://www.threepanelsoul.com/comic/on-sea-lions

deleted 10y ago

[deleted]

darkr 10y ago

This is why you should access controls on configure large mail groups. I thought this was pretty standard at large organisations. A select few people can actually send to a given group, but anyone else will just receive a bounceback.

Thriptic 10y ago

gg no re?