11 ms·
US Senate’s email system melts down in face of security test and reply-all chaos
- SoftTalker 3y agoI've seen this happen a few times, usually someone accidentally spams their entire address book, or even better a bunch of listserv addresses. Then you get the "reply all" messages to "take me off this list" or "unsubscribe" which just adds fuel to the fire.
- dessimus 3y agoThank god for Transport Rules to just silently drop emails with matching subject for a few days.
- JoeOfTexas 3y agoThis happened today at work. Like just ignore the email...
- jjoonathan 3y agoBut that's no fun! ( •̀ᴗ•́ )و
- Sylamore 3y agoOne of the privileges of previously being an Exchange admin was being able to send the "Do not reply all" message and then setting up a rule on the server to kill all further messages. But it's also why I had a script to lock down any distribution list with more than 100 members to authorized senders only.
- CatWChainsaw 3y agoThe last storm I was in had people replying all to tell people to not reply all, followed by people replying all to shame the people who replied telling people not to reply all. Followed by others replying all asking how to mute the conversation, and others replying all with instructions. Multiple times. Which prompted more people to reply all commenting on the increasing recursion of the absurdity of the whole thing. So you'd get fed up and mute the chain and think you'd think you'd killed the process, but any time someone changed the email subject line and replied all again it came back like a bad penny. Which caused more people to angrily reply all with vulgarities, which caused supervisors to reply all that their behavior was being noted and disciplinary meetings would be had. And then one day, someone replied-all to one of the child email chains with the entire script of The Bee Movie. Which launched more of all of the above. And did the same thing to every other child email chain, sporadically, for a few months. I think it took over half a year for the last reply-all to quiet down. I think at least one of those chains lives on for posterity in one of my folders, and now you tempt me...
- mensetmanusman 3y agoMS did it first: https://techcommunity.microsoft.com/t5/exchange-team-blog/me-too/ba-p/610643 https://techcommunity.microsoft.com/t5/exchange-team-blog/me...
- 8f2ab37a-ed6c 3y agoAmusing, almost 30 years later and this same problem still exists: https://www.linkedin.com/pulse/surviving-bedlam3-email-storm-brought-down-microsoft-richard-bliss/ https://www.linkedin.com/pulse/surviving-bedlam3-email-storm...
- nvy 3y agoReply-All cascades are classic entertainment in government work. When I was in the Air Force some poor Warrant Officer started a Reply-All clusterfuck when he replied to a base-wide email about issued Android phones. After about 30 Reply-Alls some very senior officer replied (to all) something to the effect of "Nobody else Reply, or your ass is grass" to which some joker from Special Operations replied (all) "Copy, Sir". Some of the most fun you can have in the public sector.
- foota 3y agoIs this a copypasta? If not, it should be.
- chatmasta 3y agoWe had this in college where someone sent "M to mute" and then thousands of people started sending emails with "M" in the body. Nobody knows how many of them were being sarcastic, but it became a meme that we found ways to insert into all sorts of unwanted group communication channels.
- rakoo 3y agoWhen the friends groupchat becomes overly active about a topic only few are interested in (not including me), I joke and reply UNSUBSCRIBE. I think I'm the only one to understand, and definitely the only one to laugh
- Jtsummers 3y ago
- eclipxe 3y agoFor Amazonians - Wallet?
- jrockway 3y agoDetails on the Wallet thing: https://news.ycombinator.com/item?id=37336088 https://news.ycombinator.com/item?id=37336088
- BLKNSLVR 3y agoBack in the days before ransomware and RATs and APTs and ubiquitous cloud-hosted corporate email, when viruses were accidentally introduced rather than spear-phished, state/criminal-gang-sponsored existential threats to a company, I had the idea (without the ability to manifest it) of a virus that would infect an MS Exchange instance and set everyone's Out of Office on and all incoming email to be forwarded to every other mailbox in the company. Not sure if it is or was feasible, but I still like the simplicity of an automation-stack doing the really hard work of exponential amplification.
- ChrisArchitect 3y agoRelated recent thread with lots of reply-all anecdotes: * @gmail.com (xkcd) https://news.ycombinator.com/item?id=37333848 https://news.ycombinator.com/item?id=37333848
- chevman 3y agoMy other favorite is when corporate comms or HR sends out some calendar invite to a massive number of people, does not use bcc, and then the server crashes when folks go in to the accepted invite on their calendar and you get thousands of calls to 'check availability' flood in. Have seen that bring the Exchange Server to its knees a few times :)
- MBCook 3y agoOr the out of office auto-replies. Heh.
- WarOnPrivacy 3y agoIn college, I came up with a one-liner to send the entire MAN to every email address. The server outage disrupted classes for a week. I can't come up with anyone who wasn't >irritated with me.
- wolverine876 3y ago> Senate staff were instructed to respond to the email, which outlined a potential threat for the drill, with their status and location on Capitol Hill or off campus. But the thousands of simultaneous reply-alls that ensued first clogged up inboxes before staff were unable to get into their inboxes altogether. That's why we test! Imagine if that happened during an actual crisis. Give them a break - that's the point of testing, to discover these glitches. Note that another negative outcome is that high value targets (e.g., Schumer and McConnell) are emailing every Senate employee with their location, not to mention the email chain also provides aggregated information on where crowds of targets can be found. Also, email ain't so secure. A sophisticated enemy - and I'm pretty sure the US government actually has some of those - would love that information. Heck, you can find locations just by locating mail traffic. > The Senate’s email system, run through Microsoft Outlook It's 2023 and someone is still writing that?
- MR4D 3y ago> Imagine if that happened during an actual crisis. I’m ironically imagining our enemies are coming up with some heinous reply-all scheme that will bring us to our knees.
- wolverine876 3y agoIt's called "Social Media".
- IggleSniggle 3y agoSome PsyOps person on here had a term for this, slowing your enemy down by getting them bogged down in unimportant details, but I've forgotten what it was...
- jjk166 3y agoSounds like a cross between bikeshedding and nerd sniping.
- 3y ago
- Someone1234 3y agoThis problem is relatively easy to solve with a Global rule: - TO: [All] - Subject: ^(?i)(re|fw):\s - Action: Drop. Heck, you could go one step further and block all TO: [ALL] entirely since they should BCC.
- tgv 3y agoThe latter is best. I think another sensible action is to remove “all” from the reply. If you really want to send to all, you have to enter it yourself.
- dataflow 3y agoIf you go this route (which isn't great), you probably want to include "Fwd" as well.
- furyg3 3y agoWay back in the day when I was doing a lot of Exchange consulting work, we would recommend removing permission to send to all except for one or two specific accounts. This was to avoid reply-all meltdowns, for security (viruses couldn't send to all), and also from an HR / internal communication perspective. We would recommend something similar for department-level groups, but often with a bit more freedom.
- kylehotchkiss 3y agoGreat way to keep busy on anything but passing any sort of funding bill!
- EMCymatics 3y ago[flagged]
- salawat 3y agoBlame the gorilla using the tool, not the tool itself. Internalize the fact you suck first, and find enlightenment.
- EMCymatics 3y agoI'll blame the gorillas making the flawed tools
- bagels 3y agoSame thing happened at Microsoft, and Facebook. I'm sure it's happened anywhere with large email lists. Servers should probably just reject replies to lists large lists.
- langsoul-com 3y agoSurely there's a better way of getting everyone's location in an emergency. Although, long/lat aren't easy enough to say, type or find.
- lapsis_beeftech 3y agoSeems trivially easy to solve since all the users are already carrying tracking devices with GPS.
- jjk166 3y agoPresumably they are more interested in information like which room someone is in than their position on Earth. I doubt that indoors the location accuracy of phone GPS would be accurate enough to be useful, particularly in buildings where many locations are designed to be secure both from physical attacks and espionage (probably a lot of thick metal plates in random walls). And even without interference issues, GPS elevation accuracy is pretty bad. Finally, you're only tracking the location of the person's phone, which in an emergency could easily get left behind somewhere. None of this is to say an email chain is the optimal solution though.
- Angostura 3y agoIf there's a cheap, simple way to implement it - I'd quite like to know.
- SturgeonsLaw 3y ago"You guys all have phones right?" The gov could write an app, installed on every senator's phone, that tracks their location as needed. Perhaps the senators might be against such invasive surveillance? Then I'd hope their voting record on such bills matches their privacy concerns when they're the ones affected.
- Angostura 3y agoThat gives location. Not status. So it tells you whether they are in the building, but not if they are fine, injured, incapacitated or unconscious. Moreover, lets say the senators have a mixed record on massive invasion of privacy. You need a system that covers everyone
- oersted 3y agoI finally got around to watching The West Wing. There's a lighthearted small subplot of exactly this in S01E19. By the way, it is so refreshing nowadays to watch a show about extremely competent imperfect people that legitimately strive to do as much good as they can, as they see it. Likewise with Sorkin's writing, while unrealistically witty and fast, it is a delight watching someone so obviously talented give their all to their work. Granted, as in other Sorkin writing, it does sound a bit like all the characters are just versions of him with different voices. But it seems to me like an intentional stylistic choice, a somewhat heightened reality.
- 22289d 3y agoThe movie Malice is one of my all-time favorites. 1993, average reviews. Total random movie. Recently I paid attention to the credits: written by Aaron Sorkin. It all makes sense now. It's been ignored for whatever reason but it's amazing.
- developer93 3y agoI call this competence porn - a brief escape to a world where leaders are both competent and benign.
- deleted 3y ago[deleted]
- saghm 3y agoOf all things happening in a West Wing episode (although to the White House, not the Senate) to still be topical 25 years after airing, this would not have been high on my list. I'm not sure to consider myself overly optimistic or overly naive...
- cvccvroomvroom 3y agoTerminate everyone who pressed reply-to-all as they can't be trusted to follow simple directions and they shouldn't have a security clearance either.
- mattbrewsbytes 3y agoAnd here I thought the oxy-moronic reply-all pavlovian response had been trained out of humans. I've been fortunate enough to work on teams where I haven't witnessed this in at least a decade. Of course its in our technically inept government where this happens.
- toyg 3y agoIt's 2023, we have systems processing trillions of transactions per minute, and still a few thousand emails can "melt" an email server? That's testament to how little consideration the mail stack gets.
- db48x 3y agoWell, kinda. Due to features like BCC headers, each recipient is supposed to get their own unique copy of every email, so email servers can end up spending a lot of time writing tens of thousands of copies of every mail sent to the list to disk. And then of course they have to sync that to all of the redundant servers, to support HA.
- breakwaterlabs 3y agoSymbolic links and deltas are a thing. None of this is hard from a technology standpoint. Exchange just isn't the best designed or most modern software.
- db48x 3y agoI don’t entirely disagree, but while deltas can be small emails are usually pretty small themselves. It might be a single 4k write to each mailbox either way. And remember that appending to a file requires the OS to do a read plus a write, while adding a file to a directory requires a write plus an update to the directory itself (another read+write at minimum). Both require allocating sectors, which means updating the freelist(s), Of course there are filesystems like ZFS that can batch up all of those writes so that they become linear writes instead of random writes, but that’s not available if you‘re running Exchange. I have no doubt that email servers could be better optimized, but only because I have that optimistic belief about all software as a general rule. (The probability that we have already found the most optimal way to write any non–trivial program you happen to examine is pretty low.) On the other hand, I doubt that Exchange has made _no_ progress at all over the decades. The fact that their email server fell over from the load probably has more to do with poor choice of hardware than any software flaw. Their Exchange server apparently fell over while sending email to “thousands” of people, while the Bedlam DL3 incident at Microsoft involved a mailing list with 13k people on it. In 1997. Maybe that order of magnitude difference means that the Senate needs to buy an NVME disk. (Of course that ignores the fact that asking thousands of people to send you their location via email is pure stupidity; this is the Senate we’re talking about.)
- TurkishPoptart 3y agohttps://en.wikipedia.org/wiki/Email_storm https://en.wikipedia.org/wiki/Email_storm
- treesknees 3y agoThis happened to us several years ago, a basic reply-all storm across probably 30k employees. The icing on the cake though - when the storm finally calmed down a few hours later, the original sender then tried to recall the message. The recall notification went out to everyone and sparked it back up again.
- kthejoker2 3y agoGot caught in 2 different bad reply all storms at Accenture a few years ago. The temptation to respond and mock / yell at the people who keep replying is strong. Feel like it should be a fireable offense at a place presumably hiring for critical thinking and situational awareness.
- deleted 3y ago[deleted]
- PeterStuer 3y agoRemember the days before mailing list distributers filtered out of office replies? It was fairly common to open your mail in the morning and have to delete several hunderds of these auto replies and dozens of replies from people complaining about it adding to the bulk.