13 ms·
Mail Loop From Hell (2012)
- brian_herman__ 8y agoI love these kind of stories this should be on bash.org
- dbrgn 8y agoThen you should check out https://dbrgn.ch/stories-from-the-internet.html https://dbrgn.ch/stories-from-the-internet.html. If you know more stories that should go on the list, let me know :)
- dTal 8y agoHoly crap, the series surrounding the 'recorder' one is the most gripping and well written tales of combined technical and political achievement I've ever read on the internet. True hacker porn. Start here: https://www.reddit.com/r/talesfromtechsupport/comments/qrqsa/ten_minutes_a_day/ https://www.reddit.com/r/talesfromtechsupport/comments/qrqsa...
- artificial 8y agoI present to you: The Worst Roommate https://forums.somethingawful.com/showthread.php?s=&threadid=1353420 https://forums.somethingawful.com/showthread.php?s=&threadid...
- brian_herman__ 8y agoYes, I read them all at work too thanks!
- krylon 8y agoAt work, we had a mail loop once; a girl from accounting had ordered toner for our printers last thing before she went on vacation and enabled an auto reply on her mailbox. Five minutes after she left the office, the order confirmation arrived, sent from noreply@random.office.supplies.whatever. Her mailbox sent an auto-reply. Our mail server sent her an error message, telling her that her reply had not been delivered. To which her mailbox happily replied with the same auto-reply mail... When I walked into the office on Monday, her mailbox had accumulated around 50,000 of those mails. Fun times.
- astrodust 8y agoI've got a client with an automated ticketing system that loves to get into mail loops with our automated ticketing system. They keep confirming that they've received a reply or update to the previous ticket and a single inquiry can rattle back and forth dozens of times.
- justjash 8y agoYep, had the same thing happen after I implemented an automated mailing process at work. Luckily the ticketing system we hit was slower and I caught it around 6k emails the next day.
- Bluecobra 8y agoIs it Jira? If so, I found out the hard way that all you need to take it down is send an emoji in your email.
- lstamour 8y agoIf said Jira server is backed my MySQL in the default “UTF8” mode that isn’t actually UTF8. Tell your Jira admins to switch to Postgres or Jira Cloud and all will be well. The same thing happens to Wordpress installs all the time, sadly.
- isodude 8y agoOr update to UTF8mb4 :)
- ancarda 8y agoThat may not be enough -- the connection also has to be utf8mb4 as well as other settings: https://mathiasbynens.be/notes/mysql-utf8mb4 https://mathiasbynens.be/notes/mysql-utf8mb4 I have no idea if JIRA supports configuring every aspect of setting up a MySQL connection.
- duxup 8y ago
- mixedmath 8y agoI love these sorts of stories. This is also one of the things that HN seems to aggregate and bubble up better than any other forum/aggregator for me. Thank you for the link, and for the post on your site.
- zentiggr 8y agoSomebody managed to find a low level PDL that referenced nearly everyone in the company globally. Just the reply-all "I'm not sure this as anything to do with me, please take me off your DL" mails forced the servers to be taken down and nearly five hours of global mail handling was wiped clean... it was about seven hours before accounts started acting normally again. Sorta amusing from the random employee view.
- saagarjha 8y agoThis sort of thing was surprisingly common at my old job. Someone would find a list that seemed relevant, ask a question on it, and not realize the list had as a sublist something like basically all of software engineering. Then people would reply-all to that asking to be taken off the list that they didn’t need to be on…
- deleted 8y ago[deleted]
- minikites 8y agoA similar story from Microsoft: https://blogs.technet.microsoft.com/exchange/2004/04/08/me-too/ https://blogs.technet.microsoft.com/exchange/2004/04/08/me-t... >So those 15,000,000 email messages collectively consumed 195,000,000,000 bytes of bandwidth. Yes, 195 gigabytes of bandwidth bouncing around between the email servers.
- bradknowles 8y agoThat’s pretty good! Almost as good as breaking all internet e-mail for the whole world in August of 1996: https://www.theregister.co.uk/2018/04/16/who_me/ https://www.theregister.co.uk/2018/04/16/who_me/
- deleted 8y ago[deleted]
- greendesk 8y agoSome years ago I run into a e-mail loop. An e-mail request is sent to a marketing e-mail service. For a specific e-mail notification, we do not receive a confirmation from the marketing e-mail service that the e-mail reached the recipient. So our service re-sends the e-mail. This goes on for several days. It turned out a filter did not let the notification reach our end. By the time we figured it out, someone received thousands of the same marketing e-mail.
- pbiggar 8y agoI remember a story from my tech support days: someone forged an email to support@eircom.net from support@indigo.ie, both of which had auto-responders.
- maaaats 8y agoNot the same kind of loop, but a few years ago I had to quickly fix email sending in some PHP service. Being rusty in PHP, I at least managed to write a for loop iterating over the list and send mails. Or so I thought. I had forgot "$" in front of the "i" in the for-header, making it never terminate, since of course that's still valid php. Luckily it was only my private email in the test-list. But I still got so many thousands of mails my Gmail account was blocked for days. Also lucky this didn't affect the delivery of our future mail (not blacklisted).
- mike-cardwell 8y agoWhen I set up an autoresponder in my .sieve file, I specify ":days n", where "n" is the minimum number of days between allowing an autoresponse to be sent to the same address. This isn't fool proof though. It would fail if the other side of the loop didn't have a similar defense, and also sent each email response from a different address.
- jtokoph 8y agoBack when I was working at an SMS related company I managed to accidentally infinite loop sending myself text messages. The aggregator and AT&T had no issues, but my iPhone became completely unresponsive to touch as I saw the notification badge quickly ramp up from 1 to tens of thousands. If I remember correctly I had to wait a few hours for the messages to be delivered and processed by the phone and then wiped my device.
- ineedasername 8y agoI once made a mistake, but it wasn't a just a loop, it was recursive. I was working in an old version of FOCUS, a reporting language, mixed with DCL. (This was a VMS environment) The details are fuzzy now, but IIRC I misplaced a single "GOTO" line of code. The result was doubly recursive: Each person on the list got the message once for every person on the list, with the content of previous recipients concatenated to the message. So: person 1 got the message once. Then person 1 & 2 got the message again, only person 2's content was appended to person 1. The person 1, 2 & 3 got the message, with 3 appended to 2, and 2 & 3 appended to 1, and so on. I was very lucky that it was an old VMS environment with low disk quotas, so my "sent" folder ballooned before the full list could run. I think I got through about 130 recipients, which came out to about 8000 messages.
- jpmoyn 8y agoThis is a great, hilarious story. Something about IRC makes for a really nice story telling platform. Thanks for sharing.
- blakesmith 8y agoIf you like the IRC format, check out http://bash.org http://bash.org - some pretty hilarious IRC logs from a glorious time in Internet history
- snazz 8y agoThere are some gems on there, like this one: #127039 +(13215)- [X] <wolf> 1. Save every Free Credit Card Offer you get, Put it in pile A <wolf> 2. Save every Free Coupon You get, put that in pile B <wolf> 3. Now open the credit card mail from pile A and find the Business Reply Mail Envelope. <wolf> 4. Take the coupons from pile B and stuff them in the envelope you hold in your hand. <wolf> 5. Drop the stuffed to the brim envelopes in your mail and walk away whistling. <wolf> I have now received two phone calls from the credit card companies telling me that they received a stuffed envelope with coupons rather then my application. They informed me that it they are not pleased that they footed the bill for the crap I sent them. I reply with "It says Business Reply Mail" I'm suggesting coupons to you to ensure that your business is more successful. They promptly hang up on me. <wolf> Now, I did this for about a month before it got boring, so I got an added idea! I added exactly 33 cents worth of pennies to the envelope so they paid EXTRA due to the weight. I got a call informing me about the money, I said it was a mistake and I demanded my change back. After yelling at the clerk and then to the supervisor they agreed to my demands and cut me a check for the money. I hold in my hand at this very moment a check from GTE Visa for exactly 33 cents. [edit]: formatting
- yerasame 8y agoOh, wow. If they have no recourse, I think I'll start doing this, too.
- jancsika 8y agoWould be fun to build a GUI app atop an event loop like that. Subject: Fwd: mousedown 52 48 Then you could trigger interesting dialogs to the user: Error: failed to receive a mouseup for 72 hours. Do you want to cancel clicking? Click one (Ok) (Cancel)
- YeGoblynQueenne 8y agoYou can still do stuff like that today. My friend broke Yahoo and gmail. Unfortunately, we didn't document the procedure very well, so the following, from my personal notes on the incident, (with handles changed to protect my friend) is all I got: How name_surname broke yahoo mail (and also gmail) ================================================== - 21/03/17 > She doesn't quite remember what she did exactly but: a) She went into nickname@yahoo.com and added another yahoo mail box- name_surname@yahoo.com. b) This didn't seem to work (?) so she went into name_surname@yahoo.com and added nickname@yahoo.com as a mailbox. c) Then she went into gmail and added nickname@yahoo.com to it d) And finally she imported mail and contacts from both yahoo accounts into gmail. > The result: in nickname@yahoo.com you can see the name_surname@yahoo.com mailbox which contains a nickname@yahoo.com mailbox, which contains an name_surname@yahoo.com mailbox, but also nickname@yahoo.com contains a mailbox called nickname@yahoo.com and name_surname@yahoo.com contains a mailbox called name_surname@yahoo.com. And in gmail, all the email imported from nickname@yahoo.com and name_surname@yahoo.com have a very long listing of all the mailboxes in which they exist, which is the two yahoo mailboxes repeated several times. That's _many_ times. > Of course, when she tried to delete the mailboxes, whereas she only had 5 messages it looked like there were 35. They were all duplicates, obviously.
- jsjohnst 8y agoIt’s hard to read on mobile due to your formatting, but based on knowledge I have of both (and lack of details in your story), I think the most your friend can claim is “broke my ymail/gmail”, not broke either service.
- YeGoblynQueenne 8y agoSorry about the formatting. Yes, she broke it for her, not everyone.
- empath75 8y agoI crashed lotus notes at a federal agency after sending a pirated dvd as an attachment. I somehow did not get fired (late ‘90s)
- lostlogin 8y agoA government agency with a file size limit above a couple of mb in the 90s?
- empath75 8y agoIt didn’t have a file size limit at all until I did that.
- bradknowles 8y agoAt a company I worked for in the Netherlands soon after the turn of the century, many of the Asian sales reps had massive Pirated DVD archives that they would offer to customers as an incentive, and then would send out those images by e-mail — across a slow E-1 leased line back to corporate HQ in Eindhoven, and then back out to their customers in Asia. Clogged the leased lines for days when they did that. That’s when I was called in to fix the email servers and implement message size limits across the board.
- ineedasername 8y agoAnother story about inappropriate emails, luckily I wasn't the perpetrator here. It was the late 90's. Email systems were generally wide open for exploitation. (It was the golden age of Spam) I myself had figured out that it was trivial to send a message and make it look like it came from anyone. Simple: telnet <server> 25 HELO MAIL FROM: <whatever you want> RCPT TO: <recipient> DATA <body contents> QUIT I used this a few times to mess with friends, send them messages from God, etc. Good times. Thing is, I wasn't the only one who'd figured this out, and the university had an "All Users" list with only 3 or 4 people whitelisted to send it messages. Someone else knew about the forged FROM scheme, and speculated that the "All Users" list didn't do any fancy verification of the "From" address. This speculation was correct. And the University President was one of those 3 or 4 people whose email address was whitelisted. The problem for this student was that they thought to themselves, "No, it couldn't be that easy..." And proceeded to send a test message. It was that easy, and their test message sent successfully. The contents were... explosive. First, it cancelled all classes for the day (of course). Had it stopped there, this story might have ended as just an ultimately harmless prank. But it went further. The message went on to describe how the president, with this free time from classes, would make herself available to provide, um, carnal knowledge of a very uncomfortable place at a particular time & location. It didn't take long. IP addresses were statically assigned. It was easy to locate the offending computer, which in this case was in a computer lab. So, cross reference login times against that IP address, right? Nope, not necessary: The computer was the one reserved only for the lab assistant on duty. Expulsion was rapid. In an interview with the student newspaper, the student decried, "I didn't think it would work!"
- umvi 8y agoReminds me of the time I was working at the Air Force Academy hospital in the basement warehouse in the mid 2000s. There were a few computers, and I wanted to mess with one of my coworkers (a senior airman) who was sitting at the desk in front of me. There's a Windows command "net send" where you can make a pop up dialogue appear on a user's computer with a custom message. The problem was, I didn't know his exact username, so I did "net send * 'What's up dog??'". The dialogue popped up on his computer, he looked confused, and I was giddy. So I went further and sent a few more, including "You suck at computers" and "Your computer has been infected with a virus, sucka". It wasn't until a few minutes later when some visibly distressed IT personnel rushed into the warehouse and started interrogating people and checking the computers one by one that I realized something was terribly, terribly wrong. Apparently the dialogues were popping up on every single computer in the entire hospital, including the commander's (a full colonel). I'm sure there would have been more disciplinary action had I not been a high school civilian intern. Everyone started calling me "Neo" from then on, but in a derogatory way.
- stcredzero 8y agoI knew immediately .forward files were involved, before I even read it. I had done something similar and brought down my college's email system back in the 80's.
- nisa 8y agoNot a loop from hell, but let's not stop the procrastination loop: The 500 miles sendmail story still fascinates me, to this day: http://web.mit.edu/jemorris/humor/500-miles http://web.mit.edu/jemorris/humor/500-miles there is even a faq: http://www.ibiblio.org/harris/500milemail-faq.html http://www.ibiblio.org/harris/500milemail-faq.html
- WeAreGoingIn 8y agoThanks for sharing! I have never laughed so much in a long time. Fantastic story.
- justwalt 8y agoI thought of this story a few weeks ago when I was using VS Code on an old machine and couldn't get the linter to work on longer files. Very similar problem and solution.
- suzzer99 8y agoWe had a Social email loop when I worked for a company with 33k employees. Someone sent out a newsletter to the company ALL mailbox, which had some tips for winterizing, and asked mostly rhetorically "What are your tips for preparing for winter?" A handful people replied to the entire company w/o realizing it - stuff like "Well I'm in California, I don't have to worry about it. Ha!". I think it would have ended there. But some grumpy programmer in NYC of course decided to reply "People. Stop replying-all!" One would think being a programmer he'd be smart enough to reply to just the people who had unintentionally spammed the whole company. But no. He spammed the whole company. And after that it was on. Dozens of emails a minute. Tons of replies like "Please take me off this list." Tons more grumpy people complaining. Big bosses pulling rank "Ok I am Vice President of muckety muck and we need to stop this replying all right now!" And the best: "Stop replying all to everyone to tell them to stop replying all!" - replied to all of course. O_o It didn't help that a lot of people worked in labs and shared computers. So they'd log on and see a ton of emails w/o knowing what happened. By the end of the week it finally died down. Then the next Monday people came back from vacation, saw 500+ emails, and of course asked to be taken off the list. This went on for weeks before it finally trickled to a halt. Why anyone was allowed to spam the entire company, or the address couldn't be blocked, I have no idea. I was tempted to use it to fish for a tennis partner. And then apologize and claim ignorance of course.
- WeAreGoingIn 8y agoWhat a hilarious story. Talking about email ethics. I remember myself doing a similar mistake, new at work. This old timer came barging into my cubicle and commanding me to “stop emailing all!”.
- gerdesj 8y agoIn the UK the National Health Service is a fair old size. A test message caused a few snags for a while: https://www.theregister.co.uk/2017/01/31/nhs_reply_all_email_fail_half_billion_messages/ https://www.theregister.co.uk/2017/01/31/nhs_reply_all_email...
- samprotas 8y agoThis exact same thing happened at DuPont about 5 years ago. I believe there were roughly 80k employees at the time. There was essentially no real work email for most of the day until I believe IT figured out how to add a whitelist for @company emails. My theory was, when they switched from LotusNotes to Outlook, they never ported over years of worth of experience encoded in user permissions. At least that fits well with the warning people always give about re-writing legacy software....
- creeble 8y agoStill easy to do, even between a couple of hosts: https://imgur.com/DwbRfDs https://imgur.com/DwbRfDs Embarassing, yes. Especially since it was ongoing for, um, months. nullmailer really likes to send mail. But some MTA's don't like to accept mail larger than a certain size. Ugh.
- lightlyused 8y agolog4net + java dev app server + debugging the db layer + database outage on a Sunday = annoyed exchange admins on a Monday.
- harikb 8y agoThe infamous Rudy Giuliani tweet needs to be added here
- tuzakey 8y agoThis reminds be of the classic Microsoft Bedlam DL3 story: https://blogs.technet.microsoft.com/exchange/2004/04/08/me-too/ https://blogs.technet.microsoft.com/exchange/2004/04/08/me-t...
- ob2 8y agoThere was a time when overnight I received 200 copies of an email in my inbox. The emails were sent manually, typos, delays and all. Very industrious individual. I was discussing this with a colleague of mine the next day and it transpired that they'd recently taken delivery of two DEC Alpha workstations. They were both sitting on his desk, being configured for actual work. We set up a system using .forward files where DEC Alpha 1 would forward an email to DEC Alpha 2. On DEC Alpha 2 we set up the same in reverse. We sent an email to DEC Alpha 1 and CC'd in our individual. We let it run for 5 seconds. In that time the two workstations, doing nothing other than sending email, managed a combined 25k of messages. We never did hear from that person again. These days when I think back about that, it was probably one of the early DOS attacks. We were a little shocked how well it worked.
- michaelcampbell 8y ago> I managed to bring down 13 CIA offices, all FBI offices east of the Mississippi, and the entire Southeastern university Research Network. I'm skeptical. Even assuming this is true, would the CIA divulge this info?
- brazzledazzle 8y agoA few years back I had mail to an address forwarding to a team at a different company. It was used to cc them on tickets, basically so they could make changes on their side when certain requests came in. At some point they decided to forward these directly into their own ticket system. Each email from our ticket system would forward and land in their own ticket system that would strip off the identifiers from our ticket system and create a new ticket which would reply to our ticket system, creating a new ticket on our side and so on. Unfortunately that queue on our side wasn’t in use by humans so no one saw the exponential mail storm building. Many mail servers have relay loop detection based on IDs in the headers but reply loops are a different beast. Once we realized what was going on it was simple enough to black hole the various addresses involved but it was pretty cool to watch our ticket and mail systems DOSing each other in real time.
- cafard 8y agoIn January 2017, NetApp had a reply-all storm. One would think that sysadms would know better than to participate, but one would not be wholly correct.