35 ms·

HBO Max accidentally sent an integration email test to users

565 points by minimaxir 5y ago

minimaxir 5y ago
Comments on Twitter: https://twitter.com/search?q=hbo%20max&src=typed_query&f=live https://twitter.com/search?q=hbo%20max&src=typed_query&f=liv...
mattsolle 5y ago
We all make mistakes but rarely at this scale. I hope whoever accidentally sent out the integration email to everyone doesn't get in trouble. I would hate for my dumb mistakes to be put on blast by hundreds of thousands or millions of people.
- dag11 5y ago
  Surely HBO will be trending on twitter within the hour. The engineer should get a raise :)
  - deleted 5y ago
    [deleted]
  - nr2x 5y ago
    a bonus equivalent to earned media exposure definitely should be considered.
  - zzyzxd 5y ago
    A lot of confused users will mark the email as a spam though. It would be interesting for the marketing team to check the related metrics before and after...
- lathiat 5y ago
  I often see "this person should be fired" etc in response to legitimate mistakes.. but that's massively undervaluing the mistake. There's a pretty good chance the person who made such a mistake is less likely to repeat it or other mistakes than someone who hasn't yet had the experience. So at best you're just sending a more valuable employee to one of your competitors. There's some gray area when it comes to making a poor judgement call to skip a well defined procedure or be lazy.. and some cases this doesn't apply to like intentional malice or a continued series of similar obviously avoidable mistakes. On a related note im often frustrated when people play a blame game (even if one might be somewhat called for, in a way) instead of considering "human factors" and how to systematically avoid people being able to make such mistakes. Or refuse to accept a human factors style explanation and just say the person "should have known better" etc. The best high profile example was that hawaii missile alert that was supposed to be a test having an incredibly poor UI. But there are countless examples everywhere all the time. I often try to think about this a lot in my work.
  - jffry 5y ago
    How's the anecdote go... "Why would I fire him? I just spend $200,000 training him not to do this thing, he'll never make this mistake again!"
    - deleted 5y ago
      [deleted]
    - edgyquant 5y ago
      Wasn’t it, “I just spent 1,000,000 on training there’s no way I could afford to fire you.”
      - jffry 5y ago
        Ah, that's I think closer to what I was remembering!
    - _kst_ 5y ago
      According to one source: “Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?” – Thomas John Watson Sr., IBM
  - collinvandyck76 5y ago
    If someone can make these kinds of mistakes, IMHO it is usually not their fault and instead is the fault of the systems that failed to prevent it from happening.
    - zzyzxd 5y ago
      It is still their fault -- they did something they shouldn't have done. But the mistake exposes a bigger problem in the underlying infra, which is a good thing.
      - jffry 5y ago
- avgDev 5y ago
  You would think there was some kind of thing in place to prevent an email like this.
  - BurningFrog 5y ago
    There usually is, after you've done this the first time.
  - edgyquant 5y ago
    Only way there wouldn’t be is if this was a high level engineer who made this mistake (I hope.)”
plasma 5y ago
I wonder if someone loaded a copy of the prod database into dev/CI.
dkokelley 5y ago
At least you know they do integration testing now. :)
busterarm 5y ago
I'm not even a subscriber and I got one.
- lscotte 5y ago
  Me too.
- nr2x 5y ago
  you should file a personal data request if you live in CA or EU! """ If you are a California resident, California Civil Code Section 1798.83 permits you to request information about our practices related to the disclosure of your personal information by certain members of the WarnerMedia family of companies to certain third parties for their direct marketing purposes. You may be able to opt out of our sharing of your personal information with unaffiliated third parties for the third parties’ direct marketing purposes in certain circumstances. Please send your request (along with your full name, email address, postal address, and the subject line labeled “Your California Privacy Rights”) by email at WMPrivacy@warnermediagroup.com. """ https://www.hbomax.com/privacy https://www.hbomax.com/privacy
- worldmerge 5y ago
  Me too. I reported it as spam.
- minimaxir 5y ago
  I just removed it from the submission title. (despite it raising more questions than answers)
- st0le 5y ago
  Same. I used to be a HBO GO Subscriber (not anymore). I guess they don't delete their records. :|
atonse 5y ago
Aww I’m feeling bad for the poor engineer who is saying “crap” a thousand times now, or is blissfully unaware that their day is about to get much worse. Once during a system upgrade we ran some scripts and they triggered emails to about 8,000 people before we realized it (would’ve been 150k people otherwise). The next day was all about clean up, sending an apology email etc. My mother had visited the next day and asked why I wasn’t hanging out and my son (6 at the time) said, totally unphased, “oh he can’t come now because he’s saying sorry to 8,000 people” Hope you get through it! These mistakes happen. Oh, and your test passed. :-)
- chrischen 5y ago
  Honestly I'd rather them not send an apology email... because then I'd have two useless emails. And in the grand scheme of things, it's literally just one more email I receive during the day.
  - cheschire 5y ago
    The first email was useful to someone! An apology email is useless to everyone.
    - diegoperini 5y ago
      I disagree, it's definitely useful. It's useless only if it feels/is dishonest. One way to avoid sounding dishonest is letting the dev team write the small apology piece.
  - smelendez 5y ago
    I think in the case the first email is too cryptic not to follow up without being unsettling to people.
    - acomjean 5y ago
      I got it. It one line and includes the word test so I just ignored it. It wasn’t scary but it wasn’t clear what it was.
mindfulplay 5y ago
It would be one hell of a marketing genius if this wasn't actually an integration test email. For the first time I actually read their damn email.
- tmp_anon_22 5y ago
  Right? And apparently there's a tracking pixel. On the other hand, if it was intentional its hard to believe there wouldn't have been a scaling issue in production.
  - jffry 5y ago
    I wouldn't read anything into the presence of tracking pixels. They plausibly would route all outbound mail through some system which instruments it with click tracking and tracking pixels, and probably also handles rate limiting, bounced delivery, unsubscribe, etc. At least, I'd consider doing something like that if my systems sent a lot of email and had to deal with a myriad of tracking and compliance needs. Plus if you're going to send an "accidental" email that gets everybody talking, you would absolutely want to use something a little less dry and clinical, because that'd get a lot more coverage outside of the usual tech circles.
- lozaning 5y ago
  It would have been a brilliant marketing move to announce another season of Silicon Valley.
  - joshxyz 5y ago
    I wish wish wish this was the case haha
  - theshrike79 5y ago
    TJ Miller is completely out and Thomas Middleditch turned out to be not so nice. Kumail is doing Marvel shit now. Silicon Valley isn't coming back.
nr2x 5y ago
typos and automation are a hell of a drug.
darkmagnus 5y ago
They should somehow tie this into one of their shows and use it as a marketing stunt.
bsharitt 5y ago
Heh, I saw that earlier and though that someone had pointed to the wrong DB in QA or dev.
throwaaskjdfh 5y ago
Anyone can make a mistake. I think it speaks to their professionalism that they test with straightforward content in the email instead of having "underpants gnome alert" or a bunch of swear words in there. We just got a peek inside HBO's technological kitchen. Seems all right.
- cgriswald 5y ago
  > We just got a peek inside HBO's technological kitchen. Seems all right. My PS4 HBOmax app won't run, won't uninstall, and won't update. The only way out I can see is reinstalling the system software. It's possible it's a PS4 issue, but none of my other software has this problem. If the PS4 was the only way I could consume my subscription, I'd unsubscribe. So, I'm not really impressed with their technological kitchen.
warmfuzzykitten 5y ago
Just got mine. I like it. Short and sweet. No ads. More like that!
- lscotte 5y ago
  You have a good point there.
anyfoo 5y ago
Now imagine it didn't just say "test" or "asdf". I still might sometimes put a funny/lighthearted twist on debugging logs that nobody outside the company would see, but I never put in swear words, condescending things, or anything else I wouldn't be okay with if it was accidentally logged in production. I fortunately did not have to make that experience myself, but I've seen too many supposedly "internal only" messages turn uninternal.
- Wheen 5y ago
  As a former HBO dev, can confirm that the majority of test cases or random debugging messages are Game of Thrones quotes.
  - dylan604 5y ago
    I guess HBO devs are too young to be able to quote The Wire.
    - edgyquant 5y ago
      401: I robs drug dealers
    - oregano 5y ago
      404: Code Not Found. Man’s Gotta Have A Code.
      - deleted 5y ago
        [deleted]
      - dylan604 5y ago
        400: WTF Did I do?
    - jagger27 5y ago
      The Sopranos too.
    - isatty 5y ago
Johnny555 5y ago
That's a good test, good coverage from end to end. If this test succeeds, then they know it will work in production too.
sethammons 5y ago
My email sending bug: I work at a large email sending company. You’ve heard of us. One of my first days on the job (years ago now), my task was to set a cron to send out emails to folks to say “hey you’re almost out of credits you should upgrade your plan.” Well there was a bug caused by a couple different problems in the system. And it turns out that the utility got stuck on a tight loop on one poor individual’s inbox. this poor yahoo recipient received over 400,000 emails from me in a matter of seconds. Yeah, i felt pretty bad there.
- brianwawok 5y ago
  I did that to myself via gmail. Gmail has some kind of weird quota. It got to the point where it would only let like 100 emails per hour through. So that 5 minutes of email blast turned into one big blast, and then 100 emails per hour, for like 14 days straight. Thanks filters at least to send them to the trash..
- bombcar 5y ago
  In a way it’s better to completely obliterate one persons inbox than sent one email by accident to 400k people. The first you could apologize in many ways including financial, the second you really can only apologize by doing it again.
  - dylan604 5y ago
    >the second you really can only apologize by doing it again. Do you though? Is there value in sending yet another email? Is there harm in not sending another email? For those of use "tuned in", we just grab the popcorn, and show sympathy for the poor chap that is having a bad day. The rest of the world probably ignores. Maybe some people report it as spam?
    - geoduck14 5y ago
      I once sent spam out to persons unknown. It was probably sent to about 4000 people - but no one could actually be sure. So to fix it, I sent an "I'm sorry email" to the following: my boss, his boss, and 50 key customers, all on BCC so they didn't know who I was "apologizing to" and left it at that.
dylan604 5y ago
I was wanting to start a pool to see how how the Test Email # got. Test #1 seems successful. I wonder what Test Email #2 will be. I know a lot can be learned in a single test, but surely, there's more to test yet.
shrike 5y ago
My first IT job was in a large call center. I was sweeping up in a data center and there was a keyboard cable stretched across a walkway. The keyboard wasn't movable (I don't remember why) so I unplugged it from the PC, swept around it, and plugged it back in. About 2 minutes later half a dozen people run into the room. Apparently the SUN workstation I had unplugged the keyboard from was a critical component of the call manager and there was a bug that forced a reboot when the keyboard was plugged in while the system was running. I had hung up on ~36,000 people. Lesson learned and I only had to keep the data center clean for another year.
- benjohnson 5y ago
  That's....not your fault. There should have been a label on the computer near the keyboard jack written by the people who knew about the problem. If you didnt make a label afterwards yourself, the next keyboard-based outage would be your fault :)
  - elwell 5y ago
    > keyboard-based outage I love it.
    - Ashanmaril 5y ago
      Reminds me of the "load-bearing poster" bit from The Simpsons
  - Talanes 5y ago
    A cover on the service critical cable stretched across a walkway wouldn't be amiss either.
- cjbprime 5y ago
  It sounds like the real lesson that needed to be learned was taught to whoever decided that a system without any fault-tolerance that reboots whenever a keyboard is plugged in was fit for production.
sporkland 5y ago
At my company I had a co-worker named Swaroop. Who decided to use swaroop@ourcompany.com for some test email address in some e2e tests. Fast forward a day, we get an email from the CEO containing a forward from a very concerned Board Member (also named swaroop and the owner of swaroop@ourcompany.com email address) asking if we had been hacked. Obviously not as embarrassing but still fun.
- saluki 5y ago
  what are the odds?
murat131 5y ago
No-blame post-mortem of this will be fun!
Xavdidtheshadow 5y ago
Years ago I worked at a company that send out a test email to all our users with some light profanity in it. We got a little (unexpectedly) good press about it and it ended up being no harm, no foul. Hopefully everyone at HBO is as nice about today's mistake! https://thenextweb.com/news/how-fetchnotes-discovered-that-cussing-at-its-users-was-good-for-business https://thenextweb.com/news/how-fetchnotes-discovered-that-c...
gmaster1440 5y ago
https://pbs.twimg.com/media/Eeeo8gRUwAAa365.jpg https://pbs.twimg.com/media/Eeeo8gRUwAAa365.jpg
bdcravens 5y ago
Worst mistake I made like this: B2B service, we billed customers, and then added white label service where we wouldn't bill our customer's end customers (they shouldn't even know we are the underlying provider). Guess who forgot to update that code and ended up sending billing emails to everyone.
malikolivier 5y ago
At a previous start-up, we did a similar mistake. This was one of the worst day of my "engineer" life. I was very young and inexperienced at that time. We were running a Ruby-On-Rails app with push notifications. We had an "on creating" life-cycle hook on a model recording an event, so that every time a new model was saved, a job sending a push notification (android or iPhone) to the appropriate user would the triggered. This would notify the users that the "event" was done. The implementation was wrong. Instead of an "on creating" hook, the push notification logic was added in an "on saving" hook. As the model was supposed to only ever be saved once (on creation), this worked for a while, and no-one ever realized. You now guess what's going to happen. Then after some time, came the day when we needed to do some data migration. Easy right? The data-migration included updating some data in the the above model in question (all rows must be updated). We run a ruby script to update the model. Everything was working in staging environment. So we release in production. The moment we release, we sent hundreds of push notifications to each of all our clients (including to our own phone, as we were using the service). Basically everyone was receiving as many push notifications as the number of "events" they completed until now. Immediately, all the phones in the office (CEO's phone, sales rep, etc.) started to ring with complaints from customers asking what was going on. The CEO was as angry as you can imagine. But no time to be angry, as everyone needed to apologize and explain the situation to all our customers. Basically the whole team was at fault for letting the problem slip through the code review. From then on, we improved code review process, and decided to include push-notifications in staging environment too.
- granshaw 5y ago
  This right here shows the danger of relying on ActiveRecord callbacks, instead of explicitly calling the notification/etc code from the caller which did the creation – potentially very impactful things can happen in a non-obvious way
Sebb767 5y ago
Since we are all sharing our bad stories: I develop on an application which does, next to other things, send out push notifications to apps via Firebase on a schedule. I was implementing a limiter for these notifications (so you can only send ~4 in a 24h window) and tested them on a local instance by generating a lot of messages. As I wanted to create a few quickly, I didn't change the date from the default, now. The actual limit was configurable, so I went and changed the config and realized I used the wrong file all along. So I switched the file, without realizing that the other file also had the actual live credentials configured (it was used for testing while developing this feature not long ago). I only realized the mistake when I had a lot of log messages along the lines of "sending push notification" at startup. Luckily for me, we had thought of rotating the secret, so no notification got through. But I had a fun five minutes while searching my phone in panic to find out how bad it really was. Hope the poor guy working at HBO reads these stories later to brighten his day a bit :)
orev 5y ago
The bigger problem I see is that they’re clearly using production data on test systems (or using the production system for testing), including PII. This is a pretty big no-no and violates many security standards. I don’t blame the tester per se, but I do blame HBO for not having a process in place that prevents this kind of thing from happening.
- mdeeks 5y ago
  Or they ran a suite of integration tests against prod as a way to verify a release was working correctly.
  - sahkopoyta 5y ago
    but... why would you do that? production is not for that one
    - kortilla 5y ago
      If you don’t test against prod your CD pipeline is missing a critical step.
      - bpicolo 5y ago
        In some systems the cost of writing and maintaining stubbed or alternate versions of things you don’t want hit by tests in prod can be pretty overbearing. Good SLOs can serve the same purpose as well for that sort of flow
- sharken 5y ago
  Exactly this, though anonymizing production data properly so it can be used as test data is very hard. Generating useful test data is much better, but it's hard to represent all the edge-cases you see in Production.
- drewmate 5y ago
  +1. The first thought I had when I saw that email was that I felt bad for the tester/dev; it's not really their fault, but they're certainly going to get at least some backlash for this. Really, it's a director/VP level issue that this kind of mistake was even possible with my email address.
stormcode 5y ago
Oof. I think there are two types of engineers in the world. Those that have done something like this, and those that will someday do something like this.
saluki 5y ago
It's not going to help if this ran on production. But I can't recommend mailtrap.io enough for catching all emails sent from staging/development. Nice for testing as well since you have an inbox you can view all emails from.
- fomine3 5y ago
  Is there any recommendation for self hosted version of mailtrap.io?
- dawnerd 5y ago
  Mailhog is nice too.
deleted 5y ago
[deleted]
johnjboren 5y ago
They just need a count of the accounts that actually pay for it.
malkia 5y ago
Just got mine :)
andrewmcwatters 5y ago
This is a great reminder to make sure your tests are phrased in such a way that if a layperson comes across it, they can ignore it, sort of in the same way we came across "This is a test of the Emergency Broadcast System."[1] And not "Big Brother Is Watching."[2] [1]: https://en.wikipedia.org/wiki/Emergency_Broadcast_System https://en.wikipedia.org/wiki/Emergency_Broadcast_System [2]: https://www.google.com/search?q=facebook+oculus+big+brother+is+watching https://www.google.com/search?q=facebook+oculus+big+brother+...
astrojams 5y ago
If this was a marketing ploy, it is genius. The amount of play this email is getting is ridiculous. Brilliant move if it was planned.
- granshaw 5y ago
  I have some amount of suspicion that it is... "Integration Test Email #1" sounds abit TOO obvious or sanitized.
balls187 5y ago
I got two of them, and I was like "Test works."
krv2021 5y ago
This is me every time I compose an email on Mailchimp. I am always petrified that I'll send to the entire mailing list rather than send out a test email. I wish these mass-emailer companies ask you to solve a captcha, puzzle, or a simple math question before you send out a mass mailer. The heart attacks ... ugh
firekvz 5y ago
Got 1st one, 3 hours ago And I just got a 2nd one (different email) just 5 minutes ago. 3 hours of email sending, still haven't notice
deleted 5y ago
[deleted]
lxchase 5y ago
I was at a company that did something like this. However, the fallout was the sending IPs and domains got put on so many email blacklists, spent months on improving emails to no avail. Our sending success rate to Gmail never recovered even after talking to an engineer who said we were on like 4 blacklists. I hope HBO's servers or ESP fare better.
clusmore 5y ago
HBO Max response on Twitter: https://twitter.com/HBOMaxHelp/status/1405712235108917249 https://twitter.com/HBOMaxHelp/status/1405712235108917249
alasdair_ 5y ago
I set up the production servers for Pokemon GO using my personal email address as the owner. When we hit 120 million users and all the servers melted, guess which address every single one of those people was told to email about the problem? It took a script running for five days to delete everything.
- jagger27 5y ago
  Oh man. I'm assuming you used your Gmail or Microsoft account where they can handle that amount of influx. Imagine if that happened on a late 90s ISP-run email account running on a single server. Ouch.
  - fy20 5y ago
    A while ago I worked on an SMS gateway and somebody had entered their own phone number in some sort of test. Somebody tested it, and there was a bug which triggered 100s of messages to be sent to their phone. This was over a decade ago, when the messages were stored on the SIM and there was a limit as to how many they could hold (something like 20). So you just fill up the limit and that's it right? Nope, the carrier helpfully buffers messages that can't be received, and they will be sent/received when there is space on the device. I can't remember how they resolved it in the end, I guess just waiting for the messages to expire (72 hours).
  - matsemann 5y ago
    I once tried to apply a hotfix in prod by opening the PHP file through ftp and modify it inline and just save. What I was fixing was the email sending logic. Since it was a long time since I had written any PHP, I forgot to add a $ before my i-variable in the loop. It still ran, though, just getting stuck with index=0 and sending email after email. Luckily my user was the first one in the db so no one else was affected. But since PHP running on a managed host isn't something one can easily "shut off" it ran until it timed out, sending thousands of email to my gmail. While Google could handle it, it ended up locking my account for a few days with an error every time I tried opening the inbox. Luckily my domain / provider didn't get blocked or spam-listed in the future.
- 5y ago
don-code 5y ago
This happened to me once at an old job. I discovered a race condition between the process that sanitized data out of the staging environment as it was (continuously) refreshed in from a production fork, and the process that sent notification e-mails during mass events. Our customers were all rather large corporate types, so a great many "Was this sent to my stores?" type messages from VPs and C-staff ensued. I luckily wasn't involved in the communications fallout from this, but it did initiate a sweeping change inside of engineering to go proactively find and correct for these sorts of issues. We had the same pattern in use throughout our platform, and really all that stood in the way of this race condition being triggered had been QA not testing at the exact same time a critical part of that refresh process was running.
mdeeks 5y ago
I once accidentally ran a load test against our password API in production and reset the password for about 100k users to random characters. Hang in their bud. Happens to us all.
technologia 5y ago
Dunno why but I’ve gotten 7 integration test emails so far
ping_pong 5y ago
Meh, it happens. Sending out test emails is about the best-case on the spectrum. I've been involved in cleaning up incidents where someone accidentally sent out discounts worth 7 figures that were meant to be for a test but it was misconfigured. That was not a good day for a bunch of people.
kuzmanov 5y ago
Came in at 8:43 PM EST, Integration Test Email #1 with the body, “This template is used by integration tests only”
NicoJuicy 5y ago
My config had an additional property: OverrideTo. This is for the test environment and emails never go to the end user. But are redirected to that value.
kgeist 5y ago
Their tweet solely blames an intern but doesn't try to explain why an intern was running tests on production data in the first place. You kind of expect big players to have proper data protection in place, but then you hear stories about interns having access to production data with passwords in plaintext.
- throwawaaarrgh 5y ago
  I've worked for a few big players. Expect that none of your data is protected unless the government will fine & punish them/individuals if they don't. Though even then, the company just forces employees to take some token training first, so later when they're compromised they can just blame some poor random employee that had no authority to change things. Did you know only a handful of US states have laws about how you can use or handle social security numbers? If the business isn't in one of those states, expect the intern to have a big list of SSNs on their laptop. Even when there is a regulation, sometimes it's intentionally violated on a regular basis, either as part of a cost/benefit analysis, or some loophole that means nobody will personally be held responsible.
cimnine 5y ago
IMO the most important lesson to learn from such incidents is that you should never put shitty content in „emails that never get sent“, because eventually they might get sent out anyway. That way you’re fuck-up is just a minor annoyance — to laugh about and tell the tales how everyone also fucked-up that one time — rather than an actual shit-storm.
- jrockway 5y ago
  This one is pretty tame. I feel like we have all been tempted to write ridiculous test data -- "Dear Valued Customer, we regret to inform you that the HBO Max Defraculator has been compromised and your credit card details have been sold to a self-aware purchasing bot. We are sorry and wish you the best." Sending that to your customers is a minor disaster. Sending "This template is to be used by integration tests only." will be ignored by everyone who isn't a software engineer.
deleted 5y ago
[deleted]
theshadowknows 5y ago
Automation is fun because when you make one mistake you actually make it millions of times inside of a few seconds :) been there!
werber 5y ago
I laughed when I got the email, knowing I’ve done the same damn thing. Seeing it here I’m more worried for the poor soul
pg5 5y ago
Someone I know who doesn't have and never has had an HBO account received one.
- beebs93 5y ago
  Yup, same here. Not sure how I was ever on their list :/
jerrysievert 5y ago
I received it. and immediately felt for the person who accidentally sent it. it happens.
jameson 5y ago
My colleague did it once. He accidentally used an incorrect API key and sent a garbage push notification like "ohai" (don't recall exactly). We were a pizza delivery startup. Few hours later, we sent another notification as an apology. "Oh Hi. Use this coupon code for 20% discount: OHAI20" Good'ol days
theoblank 5y ago
Reality is this will lose them paying subscribers, so without a doubt this is an expensive mistake regardless of who’s to blame. Large subscription businesses are very careful about when to email the segment of their subscriber base that is paying but inactive. This email just reminded a bunch of people they are paying for something but not using it, and therefore should cancel (granted, it’s unclear who it went out to). It could cause some ESP/deliverabilty problems too. On the bright side, this can serve as a reminder to us all to make sure this doesn’t happen at our companies.
- Larrikin 5y ago
  Everyone I've asked that is paying has said they got it but paid it no mind since it had no information that seemed relevant to them or they were a developer that thought it was funny. I'm actually hard pressed to even imagine the person that is cancelling their HBO max account because this single email was just too much for them.
  - deleted 5y ago
    [deleted]
  - jffry 5y ago
    I don't imagine many people will cancel HBO Max because this email was just too much, but rather because this email reminded them that they have HBO Max, and might spur them to think "I never use that so I should cancel it"
spike021 5y ago
Apparently it really was done by an "intern". >We mistakenly sent out an empty test email to a portion of our HBO Max mailing list this evening. We apologize for the inconvenience, and as the jokes pile in, yes, it was the intern. No, really. And we’re helping them through it. https://twitter.com/hbomaxhelp/status/1405712235108917249?s=12 https://twitter.com/hbomaxhelp/status/1405712235108917249?s=...
- jffry 5y ago
  This just gives me so many more questions. I hope that their apparently compassionate public stance translates into a healthy internal response that doesn't railroad the poor intern.
- bobbyi_settv 5y ago
  I guess they're going for technically correct since 100% is indeed "a portion"
whakim 5y ago
I did something very similar in my current job during my first few weeks. We had an "engineering test" script that monkeypatched the definition of "all users" (so that it returned the emails of the engineering team rather than the several hundred thousand people) before calling "send to everyone". This is obviously bad because someone might come along and change the internals of the "send to everyone" method and that person happened to be fresh-faced me. I had been tasked with making "send to everyone" a little more judicious (e.g. excluding people who hadn't opened an email in 90 days or had reported emails as spam), so I replaced the call to "all users" with some other method that calculated the recipients. And then, because I didn't totally understand how the script worked (and it seemed to be listing out some internal emails as recipients), I ran a final "engineering test" in production to see how it would look with real email content. Whoops x a few hundred thousand. I did get some pretty great responses to my apology email though.
- jannes 5y ago
  > excluding people who […] had reported emails as spam How do you find out whether a user reports you as spam? Wouldn’t that require some sort of feedback from Google, Microsoft, etc?
  - t0mas88 5y ago
    You can get that feedback from most larger email providers. A lot of email tools can report on it automatically and make sure not to email again.
  - whakim 5y ago
    In our case, Sendgrid will send a certain webhook when you attempt to send to one of these email addresses.
floatingatoll 5y ago
HBO MAX lead Andy Forssell responded: https://twitter.com/aforssell/status/1405731205274685442?s=20 https://twitter.com/aforssell/status/1405731205274685442?s=2...
pndy 5y ago
In the last August Polish internet bank mBank had two "issues": they sent out of sudden 3 notifications onto mobile devices, one even literally saying "push notification test" and the other included some of Polish diacritics [1]. Site went offline soon after these were delivered. Then few days later, database was damaged in a way some clients could see account balance and personal data of other people - that was fixed relatively fast but permanently disabled the feature of accessing recipients list in shared accounts. No idea if that was fixed - my family moved accounts elsewhere. [1] - https://niebezpiecznik.pl/wp-content/uploads/2020/08/mbank1-489x600.jpg https://niebezpiecznik.pl/wp-content/uploads/2020/08/mbank1-...
jes 5y ago
A few decades ago, I was rewriting the standard sendmail.cf that we used in the company where I worked as a sysad. I think we had about one hundred or so Sun 3/60s and 3/80s that were using it. Well, in one of my test emails, I triggered an email loop. Suddenly, all the clients were furiously mailing the mail host and queuing up additional copies as fast as they could. To make matters worse, my test mail was unprofessional in terms of the language I used. Fortunately, it was a Sunday afternoon when it happened, and the mail stayed local. But it was scary and a real pain in the neck to halt and clean up. In a future post, I’ll share how a fellow sysad and I shorted out the RG-8 coax backbone while trying to add a vampire tap. That was another kind of all night deal. :-|
iancmceachern 5y ago
I got this email!
TheBolivianNavy 5y ago
A long time ago I worked at a company that prided itself on unrestricted source code access to promote cross-group cooperation. The check-in procedure relied on an understanding of the code review and ownership policy but wasn't enforced programmatically. One day I came back from lunch to see that someone had marked an entire Perforce depot for delete. I've never scrambled so fast to try to identify this unknown user. I figured it had to be an intern and ran across the floor looking for him, only to find a poor confused intern surrounded by out of breath developers imploring him to not hit submit. Every 30s someone else ran up to try to get him to not delete the entire depot. It was the intern's first week on the job and he thought he was just deleting some local files. The event definitely inspired some access lock-downs and automated check-in procedures going forward. --- I laughed when I saw the email but was also a little pissed because I am a former, not current customer. I hope HBO Max doesn't send out an apology email. That's the equivalent of unsubscribing from an email list only to get an email confirming the unsubscription that I also have to delete.
jve 5y ago
On 30th May I received an sms from my post office at 2AM. It says that the package with some tracking number couldn't be immediatelly delivered and there will be a followup sms when I will be able to claim that package at the local post office. I asked at the post office about it and they said: "oh, many people ask about it. We received that sms too. That is to be ignored." I wonder why they couldn't send followup sms and save people trip/call to local post office.
- notimetorelax 5y ago
  I’m assuming folks that responded to your call are not the same folks who can send the follow up SMS. Happens all too frequently when e.g. developers are detached from production and oncallers are dying under the pager load.
  - jve 5y ago
    Well, those employees should pass that info UP the chain and let them know that this mistake causes inconvenience and confusion and they should clean up the mess. Ofcourse that lady is not the one sending sms... I wonder if they are testing new functionality - I'v read that we will be able to have our own pickup station right at my house, for some subscription price or something like that. So packages/postage will come delivered directly to my door behind a closed box (given it is being delivered by national post office).
wyldfire 5y ago
Aside: I'm a big fan of HBO's content but their app is really poor quality. Playback frequently is unable to start or halts midstream. I've noticed this on distinct devices (android, AppleTV). DRM run amok? I don't know but you would think making playback work consistently would be near the top of their list. It's gotten slightly better over the last year or so. But it still happens way more often than it should.
hyperman1 5y ago
I heard how a company in the long past got a phone call from an elder customer: I've got such a strange letter from you. Reception asks what it says. Says the customer: I don't know yet, they're still busy putting it in the hallway. Huh? It turns out an error was made in a mass mailing, and every letter was sent with the first address on the list. The list was roughly sorted by birth date, so the eldest customer got all of them. Postal office workers drove with a van to the customer, and duly delivered bags and bags of mail to the same address.
- century19 5y ago
  Same thing happened somewhere I worked, except all mail was being sent to the company HQ. It was loaded on lorries but someone at the sorting centre had the foresight to phone up and ask "is this correct"?
theshrike79 5y ago
Bah, amateurs. I did this with physical mail. I was developing an e-letter integration. Basically I had to push a specifically formatted text file to an FTP server, from where the Post office's systems would grab it, print it and mail the physical letter. There was a specific "this is a test" bit that had to be set so that the actual letter wouldn't get sent. So... I kinda forgot that bit at one point and received a hefty pile of letters addressed to me to the office.
dataviz1000 5y ago
The email is innocuous. I opened the headers to see what was going on and why it got past spam blockers. My next thought was how much more valuable this mistake is than most their advertising budget to get attention.
- deleted 5y ago
  [deleted]
- deleted 5y ago
  [deleted]
herunan 5y ago
I work at WarnerMedia, who own HBO Max, and I thought this was a test using an employee account pool. Never mind, then!
aequitas 5y ago
This is why you always use the special use[0] domain names like example.com for your tests or documentation. [0] https://en.wikipedia.org/wiki/Special-use_domain_name https://en.wikipedia.org/wiki/Special-use_domain_name
makach 5y ago
These things happens. It's not your fault but a consequence of things that can improve. You'll recover. Misconfiguration is also cyber securiy threat #1 these days.
deleted 5y ago
[deleted]
callamdelaney 5y ago
It was the intern! Oops. Source: https://twitter.com/HBOMaxHelp/status/1405712235108917249 https://twitter.com/HBOMaxHelp/status/1405712235108917249
starfox64_ 5y ago
This reminds me of when I took over a project after a developer that left they company. They had just added SMS notifications to production but to my horror left some test code that would text you "Hey motherf**r" every time you would change your phone number. To this day I still wonder how many people got that text.
pachico 5y ago
I bet what happened was that their integration environment has unredacted production emails.
bowlingx 5y ago
Yap, been there :/.
beshur 5y ago
Well, this is a true integration test, when you finally reach real users :)
treyfitty 5y ago
Apparently, it was by an intern https://twitter.com/HBOMaxHelp/status/1405712235108917249?s=20 https://twitter.com/HBOMaxHelp/status/1405712235108917249?s=...
DangerousPie 5y ago
Interesting that it appears to have gone out to former subscribers too. They really shouldn't be retaining those email addresses, should they?
- lm28469 5y ago
  Are you protected by GDPR ? If so yes, if not it's just business as usual. Gather as much data and keep them just in case you need them in an hypothetical future for an hypothetical use case
jakub_g 5y ago
patio11 shared on Twitter how he pushed a faulty code to prod *just before changing apartments* so that it started sending email, sms and calling hundreds of people every 5 min for 13 hours: https://twitter.com/patio11/status/1405704339969220615 https://twitter.com/patio11/status/1405704339969220615 https://twitter.com/patio11/status/1405706868110987269 https://twitter.com/patio11/status/1405706868110987269 https://twitter.com/patio11/status/1405707285268103169 https://twitter.com/patio11/status/1405707285268103169
rojcyk 5y ago
When I was working at Kiwi.com as a senior product designer one of the guys on the development team did something very similar. We were testing a new push notification refactoring and managed to send a push notification "Test test" to our whole Android user base (couple of hundred thousands of users). Interestingly enough, it was a great reactivation campaign. We bumped bookings by a noticeable margin, and since we redesigned our "rate us" flow couple of weeks before, we had a huge influx of 5* ratings as well.
sackerhews 5y ago
Stuff like this can unfortunately happen. This is a reminder not to use use obscenities or jokes in your test data. Remember Mr. Rich Bastard: https://www.snopes.com/fact-check/dear-rich-bastard/ https://www.snopes.com/fact-check/dear-rich-bastard/
ElViajero 5y ago
> "This template is used by integration test only." No, it is not only for integration test. It is also used in production. They need to update that description. :)
napolux 5y ago
We once sent 1 single user 100k emails. We sent him flowers to apologize :P
diveanon 5y ago
Pretty sure this will happen to every web dev at some point in their career. It's why I strictly enforce professional test messages and test users, nothing worse that doing a product demo with users named Donald Trump and Osama Bin Laden. Hope the guy didn't get fired, things like this happen because of multiple failures in dev ops, not just 1 guy pressing the wrong button.
tottenhm 5y ago
I can’t wait to watch Integration Test Email #2 at the same time next week.
fleddr 5y ago
Amateur mistake, compared to my professional mistakes. I once wrote a bot that sent email within the company (100K+ employees). I kick-started the bot on a server remotely and only then discovered it's an endless loop. It required server admin rights to stop it, which I did not have. I couldn't immediately reach server admins so had to physically drive there. An hour or so later somebody helped me kill the process. The emails already sent could not be cleared out server-side, which meant that recipients had freezing email clients for over a week, unable to handle the volume, a typical 300K new emails per recipient. They had to Ctrl+A and Delete 100 emails, do this for the next 100, and so on, whilst not deleting real and useful emails. I pretty much destroyed email for those people. I don't just destroy things at scale though, also at home. Around the time we had our first home broadband internet connection, I set up a web server and just kept my PC running. Unknown to me, the web server software included an email server with open relay enabled by default. About 3 days later, my dad complained about the internet connection not working. The IPS detected the issue (millions of emails sent out via my home server) and gave us a red card, fully shutting us down, permanently.
- colejohnson66 5y ago
  To avoid accidentally deleting useful emails while “Ctrl-A-ing”, could they have run a search by sender or some other criteria first? Then only delete the search results.
  - fleddr 5y ago
    In the email client? Probably yes, if they know how. This was a very long time ago, early 2000s, and the email client was the dreaded Lotus Notes software.
    - jimmaswell 5y ago
      I had to use that client recently. I can't believe it doesn't even have a proper email search function. Best you can get is ctrl-f on the open page or sorting all your email alphabetically.
- 5y ago
nofinator 5y ago
For about 24 hours last summer, Citizen app on Android would play the audio from a "Who's that Pokemon? Rick Roll" video when there was an alert. They quietly fixed it. I can only imagine that it went to prod because it was a goofy test alert sound that accidentally got promoted out of a QA environment?
joren- 5y ago
To share my mistakes as well: After launching a web app we noticed severe performance issues and had to roll back to a previous version. To identify the performance problems introduced in the new version, I wrote a script mimicking a thousands of typical user interactions. This also included sending email. In the moment, we forgot to properly configure the dev environment and the emails went out.. but it did not stop there. This is a good decade ago and some clients still preferred to get faxes instead of emails. We used a service which automatically converted emails to fax messages. We tried to intervene but were not able to stop thousands of fax messages and basically DDOS'd many fax machines and then had to call the affected clients..
- adius 5y ago
  Maybe this prompted some people to switch to email? Thanks for your service in that case! =P
mcintyre1994 5y ago
Since we're sharing email sending mistakes: We wanted to start sending users occasional summary emails about activity on the platform that they might be interested in. We wrote the email generation logic first, and then the plan was to start adding different activity events and dogfooding the emails internally to see how they looked in practice. This would be fine because while we developed only internal users would have activity data, so only we'd get emailed. But there was a bug in the "is this email empty" logic so we once sent every user an email with the subject of something like "Check out your recent activity on X" with a few headings and absolutely no activity content. Oops.
SomeHacker44 5y ago
This is a perfect mistake. Embarrassing but mostly harmless. Probably not too expensive (depends on ESP cost). Great teachable moment. I wish I could get my engineers to make more mistakes like this. :)
detaro 5y ago
And this is why you should be careful with "funny" test messages/placeholder names/dummy products etc. This could be a lot more trouble if it weren't obviously a technical thing, although a clear sentence along the lines of "if you are a customer and received this email, sorry, please ignore it" would maybe be a slight improvement.
yaur 5y ago
Not much to add to this, except that I also got this email sent to an address which I have only given out to Hulu.
jedisct1 5y ago
Intern or not, everybody makes mistakes. Are we talking about heart surgery here? No, we're just talking about an insignificant test email. Nobody was hurt, and it reactions were actually fun to read. So, whoever unintentionally did that: this is okay. And if your manager says it's not, this is not okay.
at_a_remove 5y ago
I got this email, shrugged, deleted it, moved on. No harm no foul.
notacoward 5y ago
As is usually the case with such things, this wasn't really the intern's fault. The question I'd be asking is: why wasn't there proper access control in place to stop this? The system actually sending the emails should not have allowed "testuser" to send to a production list. So, second question: when this was proposed during some planning meeting (and it almost certainly was), why was it shot down? That's the process problem right there.
lh15 5y ago
I once wrote an endless loop triggering thousands of Twilio SMS messages. Took a few minutes to kill the server

565 points by minimaxir 5y ago

Comments on Twitter: https://twitter.com/search?q=hbo%20max&src=typed_query&f=live https://twitter.com/search?q=hbo%20max&src=typed_query&f=liv...

mattsolle 5y ago

We all make mistakes but rarely at this scale. I hope whoever accidentally sent out the integration email to everyone doesn't get in trouble. I would hate for my dumb mistakes to be put on blast by hundreds of thousands or millions of people.

dag11 5y ago
Surely HBO will be trending on twitter within the hour. The engineer should get a raise :)
- deleted 5y ago
  [deleted]
- nr2x 5y ago
  a bonus equivalent to earned media exposure definitely should be considered.
- zzyzxd 5y ago
  A lot of confused users will mark the email as a spam though. It would be interesting for the marketing team to check the related metrics before and after...
lathiat 5y ago
I often see "this person should be fired" etc in response to legitimate mistakes.. but that's massively undervaluing the mistake. There's a pretty good chance the person who made such a mistake is less likely to repeat it or other mistakes than someone who hasn't yet had the experience. So at best you're just sending a more valuable employee to one of your competitors. There's some gray area when it comes to making a poor judgement call to skip a well defined procedure or be lazy.. and some cases this doesn't apply to like intentional malice or a continued series of similar obviously avoidable mistakes. On a related note im often frustrated when people play a blame game (even if one might be somewhat called for, in a way) instead of considering "human factors" and how to systematically avoid people being able to make such mistakes. Or refuse to accept a human factors style explanation and just say the person "should have known better" etc. The best high profile example was that hawaii missile alert that was supposed to be a test having an incredibly poor UI. But there are countless examples everywhere all the time. I often try to think about this a lot in my work.
- jffry 5y ago
  How's the anecdote go... "Why would I fire him? I just spend $200,000 training him not to do this thing, he'll never make this mistake again!"
  - deleted 5y ago
    [deleted]
  - edgyquant 5y ago
    Wasn’t it, “I just spent 1,000,000 on training there’s no way I could afford to fire you.”
    - jffry 5y ago
      Ah, that's I think closer to what I was remembering!
  - _kst_ 5y ago
    According to one source: “Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?” – Thomas John Watson Sr., IBM
- collinvandyck76 5y ago
  If someone can make these kinds of mistakes, IMHO it is usually not their fault and instead is the fault of the systems that failed to prevent it from happening.
  - zzyzxd 5y ago
    It is still their fault -- they did something they shouldn't have done. But the mistake exposes a bigger problem in the underlying infra, which is a good thing.
    - jffry 5y ago
avgDev 5y ago
You would think there was some kind of thing in place to prevent an email like this.
- BurningFrog 5y ago
  There usually is, after you've done this the first time.
- edgyquant 5y ago
  Only way there wouldn’t be is if this was a high level engineer who made this mistake (I hope.)”

plasma 5y ago

I wonder if someone loaded a copy of the prod database into dev/CI.

dkokelley 5y ago

At least you know they do integration testing now. :)

busterarm 5y ago

I'm not even a subscriber and I got one.

lscotte 5y ago
Me too.
nr2x 5y ago
you should file a personal data request if you live in CA or EU! """ If you are a California resident, California Civil Code Section 1798.83 permits you to request information about our practices related to the disclosure of your personal information by certain members of the WarnerMedia family of companies to certain third parties for their direct marketing purposes. You may be able to opt out of our sharing of your personal information with unaffiliated third parties for the third parties’ direct marketing purposes in certain circumstances. Please send your request (along with your full name, email address, postal address, and the subject line labeled “Your California Privacy Rights”) by email at WMPrivacy@warnermediagroup.com. """ https://www.hbomax.com/privacy https://www.hbomax.com/privacy
worldmerge 5y ago
Me too. I reported it as spam.
minimaxir 5y ago
I just removed it from the submission title. (despite it raising more questions than answers)
st0le 5y ago
Same. I used to be a HBO GO Subscriber (not anymore). I guess they don't delete their records. :|

atonse 5y ago

Aww I’m feeling bad for the poor engineer who is saying “crap” a thousand times now, or is blissfully unaware that their day is about to get much worse. Once during a system upgrade we ran some scripts and they triggered emails to about 8,000 people before we realized it (would’ve been 150k people otherwise). The next day was all about clean up, sending an apology email etc. My mother had visited the next day and asked why I wasn’t hanging out and my son (6 at the time) said, totally unphased, “oh he can’t come now because he’s saying sorry to 8,000 people” Hope you get through it! These mistakes happen. Oh, and your test passed. :-)

chrischen 5y ago
Honestly I'd rather them not send an apology email... because then I'd have two useless emails. And in the grand scheme of things, it's literally just one more email I receive during the day.
- cheschire 5y ago
  The first email was useful to someone! An apology email is useless to everyone.
  - diegoperini 5y ago
    I disagree, it's definitely useful. It's useless only if it feels/is dishonest. One way to avoid sounding dishonest is letting the dev team write the small apology piece.
- smelendez 5y ago
  I think in the case the first email is too cryptic not to follow up without being unsettling to people.
  - acomjean 5y ago
    I got it. It one line and includes the word test so I just ignored it. It wasn’t scary but it wasn’t clear what it was.

mindfulplay 5y ago

It would be one hell of a marketing genius if this wasn't actually an integration test email. For the first time I actually read their damn email.

tmp_anon_22 5y ago
Right? And apparently there's a tracking pixel. On the other hand, if it was intentional its hard to believe there wouldn't have been a scaling issue in production.
- jffry 5y ago
  I wouldn't read anything into the presence of tracking pixels. They plausibly would route all outbound mail through some system which instruments it with click tracking and tracking pixels, and probably also handles rate limiting, bounced delivery, unsubscribe, etc. At least, I'd consider doing something like that if my systems sent a lot of email and had to deal with a myriad of tracking and compliance needs. Plus if you're going to send an "accidental" email that gets everybody talking, you would absolutely want to use something a little less dry and clinical, because that'd get a lot more coverage outside of the usual tech circles.
lozaning 5y ago
It would have been a brilliant marketing move to announce another season of Silicon Valley.
- joshxyz 5y ago
  I wish wish wish this was the case haha
- theshrike79 5y ago
  TJ Miller is completely out and Thomas Middleditch turned out to be not so nice. Kumail is doing Marvel shit now. Silicon Valley isn't coming back.

nr2x 5y ago

typos and automation are a hell of a drug.

darkmagnus 5y ago

They should somehow tie this into one of their shows and use it as a marketing stunt.

bsharitt 5y ago

Heh, I saw that earlier and though that someone had pointed to the wrong DB in QA or dev.

throwaaskjdfh 5y ago

Anyone can make a mistake. I think it speaks to their professionalism that they test with straightforward content in the email instead of having "underpants gnome alert" or a bunch of swear words in there. We just got a peek inside HBO's technological kitchen. Seems all right.

cgriswald 5y ago
> We just got a peek inside HBO's technological kitchen. Seems all right. My PS4 HBOmax app won't run, won't uninstall, and won't update. The only way out I can see is reinstalling the system software. It's possible it's a PS4 issue, but none of my other software has this problem. If the PS4 was the only way I could consume my subscription, I'd unsubscribe. So, I'm not really impressed with their technological kitchen.

warmfuzzykitten 5y ago

Just got mine. I like it. Short and sweet. No ads. More like that!

lscotte 5y ago
You have a good point there.

anyfoo 5y ago

Now imagine it didn't just say "test" or "asdf". I still might sometimes put a funny/lighthearted twist on debugging logs that nobody outside the company would see, but I never put in swear words, condescending things, or anything else I wouldn't be okay with if it was accidentally logged in production. I fortunately did not have to make that experience myself, but I've seen too many supposedly "internal only" messages turn uninternal.

Wheen 5y ago
As a former HBO dev, can confirm that the majority of test cases or random debugging messages are Game of Thrones quotes.
- dylan604 5y ago
  I guess HBO devs are too young to be able to quote The Wire.
  - edgyquant 5y ago
    401: I robs drug dealers
  - oregano 5y ago
    404: Code Not Found. Man’s Gotta Have A Code.
    - deleted 5y ago
      [deleted]
    - dylan604 5y ago
      400: WTF Did I do?
  - jagger27 5y ago
    The Sopranos too.
  - isatty 5y ago

Johnny555 5y ago

That's a good test, good coverage from end to end. If this test succeeds, then they know it will work in production too.

sethammons 5y ago

My email sending bug: I work at a large email sending company. You’ve heard of us. One of my first days on the job (years ago now), my task was to set a cron to send out emails to folks to say “hey you’re almost out of credits you should upgrade your plan.” Well there was a bug caused by a couple different problems in the system. And it turns out that the utility got stuck on a tight loop on one poor individual’s inbox. this poor yahoo recipient received over 400,000 emails from me in a matter of seconds. Yeah, i felt pretty bad there.

brianwawok 5y ago
I did that to myself via gmail. Gmail has some kind of weird quota. It got to the point where it would only let like 100 emails per hour through. So that 5 minutes of email blast turned into one big blast, and then 100 emails per hour, for like 14 days straight. Thanks filters at least to send them to the trash..
bombcar 5y ago
In a way it’s better to completely obliterate one persons inbox than sent one email by accident to 400k people. The first you could apologize in many ways including financial, the second you really can only apologize by doing it again.
- dylan604 5y ago
  >the second you really can only apologize by doing it again. Do you though? Is there value in sending yet another email? Is there harm in not sending another email? For those of use "tuned in", we just grab the popcorn, and show sympathy for the poor chap that is having a bad day. The rest of the world probably ignores. Maybe some people report it as spam?
  - geoduck14 5y ago
    I once sent spam out to persons unknown. It was probably sent to about 4000 people - but no one could actually be sure. So to fix it, I sent an "I'm sorry email" to the following: my boss, his boss, and 50 key customers, all on BCC so they didn't know who I was "apologizing to" and left it at that.

dylan604 5y ago

I was wanting to start a pool to see how how the Test Email # got. Test #1 seems successful. I wonder what Test Email #2 will be. I know a lot can be learned in a single test, but surely, there's more to test yet.

shrike 5y ago

My first IT job was in a large call center. I was sweeping up in a data center and there was a keyboard cable stretched across a walkway. The keyboard wasn't movable (I don't remember why) so I unplugged it from the PC, swept around it, and plugged it back in. About 2 minutes later half a dozen people run into the room. Apparently the SUN workstation I had unplugged the keyboard from was a critical component of the call manager and there was a bug that forced a reboot when the keyboard was plugged in while the system was running. I had hung up on ~36,000 people. Lesson learned and I only had to keep the data center clean for another year.

benjohnson 5y ago
That's....not your fault. There should have been a label on the computer near the keyboard jack written by the people who knew about the problem. If you didnt make a label afterwards yourself, the next keyboard-based outage would be your fault :)
- elwell 5y ago
  > keyboard-based outage I love it.
  - Ashanmaril 5y ago
    Reminds me of the "load-bearing poster" bit from The Simpsons
- Talanes 5y ago
  A cover on the service critical cable stretched across a walkway wouldn't be amiss either.
cjbprime 5y ago
It sounds like the real lesson that needed to be learned was taught to whoever decided that a system without any fault-tolerance that reboots whenever a keyboard is plugged in was fit for production.

sporkland 5y ago

At my company I had a co-worker named Swaroop. Who decided to use swaroop@ourcompany.com for some test email address in some e2e tests. Fast forward a day, we get an email from the CEO containing a forward from a very concerned Board Member (also named swaroop and the owner of swaroop@ourcompany.com email address) asking if we had been hacked. Obviously not as embarrassing but still fun.

saluki 5y ago
what are the odds?

murat131 5y ago

No-blame post-mortem of this will be fun!

Xavdidtheshadow 5y ago

Years ago I worked at a company that send out a test email to all our users with some light profanity in it. We got a little (unexpectedly) good press about it and it ended up being no harm, no foul. Hopefully everyone at HBO is as nice about today's mistake! https://thenextweb.com/news/how-fetchnotes-discovered-that-cussing-at-its-users-was-good-for-business https://thenextweb.com/news/how-fetchnotes-discovered-that-c...

gmaster1440 5y ago

https://pbs.twimg.com/media/Eeeo8gRUwAAa365.jpg https://pbs.twimg.com/media/Eeeo8gRUwAAa365.jpg

bdcravens 5y ago

Worst mistake I made like this: B2B service, we billed customers, and then added white label service where we wouldn't bill our customer's end customers (they shouldn't even know we are the underlying provider). Guess who forgot to update that code and ended up sending billing emails to everyone.

malikolivier 5y ago

At a previous start-up, we did a similar mistake. This was one of the worst day of my "engineer" life. I was very young and inexperienced at that time. We were running a Ruby-On-Rails app with push notifications. We had an "on creating" life-cycle hook on a model recording an event, so that every time a new model was saved, a job sending a push notification (android or iPhone) to the appropriate user would the triggered. This would notify the users that the "event" was done. The implementation was wrong. Instead of an "on creating" hook, the push notification logic was added in an "on saving" hook. As the model was supposed to only ever be saved once (on creation), this worked for a while, and no-one ever realized. You now guess what's going to happen. Then after some time, came the day when we needed to do some data migration. Easy right? The data-migration included updating some data in the the above model in question (all rows must be updated). We run a ruby script to update the model. Everything was working in staging environment. So we release in production. The moment we release, we sent hundreds of push notifications to each of all our clients (including to our own phone, as we were using the service). Basically everyone was receiving as many push notifications as the number of "events" they completed until now. Immediately, all the phones in the office (CEO's phone, sales rep, etc.) started to ring with complaints from customers asking what was going on. The CEO was as angry as you can imagine. But no time to be angry, as everyone needed to apologize and explain the situation to all our customers. Basically the whole team was at fault for letting the problem slip through the code review. From then on, we improved code review process, and decided to include push-notifications in staging environment too.

granshaw 5y ago
This right here shows the danger of relying on ActiveRecord callbacks, instead of explicitly calling the notification/etc code from the caller which did the creation – potentially very impactful things can happen in a non-obvious way

Sebb767 5y ago

Since we are all sharing our bad stories: I develop on an application which does, next to other things, send out push notifications to apps via Firebase on a schedule. I was implementing a limiter for these notifications (so you can only send ~4 in a 24h window) and tested them on a local instance by generating a lot of messages. As I wanted to create a few quickly, I didn't change the date from the default, now. The actual limit was configurable, so I went and changed the config and realized I used the wrong file all along. So I switched the file, without realizing that the other file also had the actual live credentials configured (it was used for testing while developing this feature not long ago). I only realized the mistake when I had a lot of log messages along the lines of "sending push notification" at startup. Luckily for me, we had thought of rotating the secret, so no notification got through. But I had a fun five minutes while searching my phone in panic to find out how bad it really was. Hope the poor guy working at HBO reads these stories later to brighten his day a bit :)

orev 5y ago

The bigger problem I see is that they’re clearly using production data on test systems (or using the production system for testing), including PII. This is a pretty big no-no and violates many security standards. I don’t blame the tester per se, but I do blame HBO for not having a process in place that prevents this kind of thing from happening.

mdeeks 5y ago
Or they ran a suite of integration tests against prod as a way to verify a release was working correctly.
- sahkopoyta 5y ago
  but... why would you do that? production is not for that one
  - kortilla 5y ago
    If you don’t test against prod your CD pipeline is missing a critical step.
    - bpicolo 5y ago
      In some systems the cost of writing and maintaining stubbed or alternate versions of things you don’t want hit by tests in prod can be pretty overbearing. Good SLOs can serve the same purpose as well for that sort of flow
sharken 5y ago
Exactly this, though anonymizing production data properly so it can be used as test data is very hard. Generating useful test data is much better, but it's hard to represent all the edge-cases you see in Production.
drewmate 5y ago
+1. The first thought I had when I saw that email was that I felt bad for the tester/dev; it's not really their fault, but they're certainly going to get at least some backlash for this. Really, it's a director/VP level issue that this kind of mistake was even possible with my email address.

stormcode 5y ago

Oof. I think there are two types of engineers in the world. Those that have done something like this, and those that will someday do something like this.

saluki 5y ago

It's not going to help if this ran on production. But I can't recommend mailtrap.io enough for catching all emails sent from staging/development. Nice for testing as well since you have an inbox you can view all emails from.

fomine3 5y ago
Is there any recommendation for self hosted version of mailtrap.io?
dawnerd 5y ago
Mailhog is nice too.

deleted 5y ago

[deleted]

johnjboren 5y ago

They just need a count of the accounts that actually pay for it.

malkia 5y ago

Just got mine :)

andrewmcwatters 5y ago

This is a great reminder to make sure your tests are phrased in such a way that if a layperson comes across it, they can ignore it, sort of in the same way we came across "This is a test of the Emergency Broadcast System."[1] And not "Big Brother Is Watching."[2] [1]: https://en.wikipedia.org/wiki/Emergency_Broadcast_System https://en.wikipedia.org/wiki/Emergency_Broadcast_System [2]: https://www.google.com/search?q=facebook+oculus+big+brother+is+watching https://www.google.com/search?q=facebook+oculus+big+brother+...

astrojams 5y ago

If this was a marketing ploy, it is genius. The amount of play this email is getting is ridiculous. Brilliant move if it was planned.

granshaw 5y ago
I have some amount of suspicion that it is... "Integration Test Email #1" sounds abit TOO obvious or sanitized.

balls187 5y ago

I got two of them, and I was like "Test works."

krv2021 5y ago

This is me every time I compose an email on Mailchimp. I am always petrified that I'll send to the entire mailing list rather than send out a test email. I wish these mass-emailer companies ask you to solve a captcha, puzzle, or a simple math question before you send out a mass mailer. The heart attacks ... ugh

firekvz 5y ago

Got 1st one, 3 hours ago And I just got a 2nd one (different email) just 5 minutes ago. 3 hours of email sending, still haven't notice

deleted 5y ago

[deleted]

lxchase 5y ago

I was at a company that did something like this. However, the fallout was the sending IPs and domains got put on so many email blacklists, spent months on improving emails to no avail. Our sending success rate to Gmail never recovered even after talking to an engineer who said we were on like 4 blacklists. I hope HBO's servers or ESP fare better.

clusmore 5y ago

HBO Max response on Twitter: https://twitter.com/HBOMaxHelp/status/1405712235108917249 https://twitter.com/HBOMaxHelp/status/1405712235108917249

alasdair_ 5y ago

I set up the production servers for Pokemon GO using my personal email address as the owner. When we hit 120 million users and all the servers melted, guess which address every single one of those people was told to email about the problem? It took a script running for five days to delete everything.

jagger27 5y ago
Oh man. I'm assuming you used your Gmail or Microsoft account where they can handle that amount of influx. Imagine if that happened on a late 90s ISP-run email account running on a single server. Ouch.
- fy20 5y ago
  A while ago I worked on an SMS gateway and somebody had entered their own phone number in some sort of test. Somebody tested it, and there was a bug which triggered 100s of messages to be sent to their phone. This was over a decade ago, when the messages were stored on the SIM and there was a limit as to how many they could hold (something like 20). So you just fill up the limit and that's it right? Nope, the carrier helpfully buffers messages that can't be received, and they will be sent/received when there is space on the device. I can't remember how they resolved it in the end, I guess just waiting for the messages to expire (72 hours).
- matsemann 5y ago
  I once tried to apply a hotfix in prod by opening the PHP file through ftp and modify it inline and just save. What I was fixing was the email sending logic. Since it was a long time since I had written any PHP, I forgot to add a $ before my i-variable in the loop. It still ran, though, just getting stuck with index=0 and sending email after email. Luckily my user was the first one in the db so no one else was affected. But since PHP running on a managed host isn't something one can easily "shut off" it ran until it timed out, sending thousands of email to my gmail. While Google could handle it, it ended up locking my account for a few days with an error every time I tried opening the inbox. Luckily my domain / provider didn't get blocked or spam-listed in the future.
5y ago

don-code 5y ago

This happened to me once at an old job. I discovered a race condition between the process that sanitized data out of the staging environment as it was (continuously) refreshed in from a production fork, and the process that sent notification e-mails during mass events. Our customers were all rather large corporate types, so a great many "Was this sent to my stores?" type messages from VPs and C-staff ensued. I luckily wasn't involved in the communications fallout from this, but it did initiate a sweeping change inside of engineering to go proactively find and correct for these sorts of issues. We had the same pattern in use throughout our platform, and really all that stood in the way of this race condition being triggered had been QA not testing at the exact same time a critical part of that refresh process was running.

mdeeks 5y ago

I once accidentally ran a load test against our password API in production and reset the password for about 100k users to random characters. Hang in their bud. Happens to us all.

technologia 5y ago

Dunno why but I’ve gotten 7 integration test emails so far

ping_pong 5y ago

Meh, it happens. Sending out test emails is about the best-case on the spectrum. I've been involved in cleaning up incidents where someone accidentally sent out discounts worth 7 figures that were meant to be for a test but it was misconfigured. That was not a good day for a bunch of people.

kuzmanov 5y ago

Came in at 8:43 PM EST, Integration Test Email #1 with the body, “This template is used by integration tests only”

NicoJuicy 5y ago

My config had an additional property: OverrideTo. This is for the test environment and emails never go to the end user. But are redirected to that value.

kgeist 5y ago

Their tweet solely blames an intern but doesn't try to explain why an intern was running tests on production data in the first place. You kind of expect big players to have proper data protection in place, but then you hear stories about interns having access to production data with passwords in plaintext.

throwawaaarrgh 5y ago
I've worked for a few big players. Expect that none of your data is protected unless the government will fine & punish them/individuals if they don't. Though even then, the company just forces employees to take some token training first, so later when they're compromised they can just blame some poor random employee that had no authority to change things. Did you know only a handful of US states have laws about how you can use or handle social security numbers? If the business isn't in one of those states, expect the intern to have a big list of SSNs on their laptop. Even when there is a regulation, sometimes it's intentionally violated on a regular basis, either as part of a cost/benefit analysis, or some loophole that means nobody will personally be held responsible.

cimnine 5y ago

IMO the most important lesson to learn from such incidents is that you should never put shitty content in „emails that never get sent“, because eventually they might get sent out anyway. That way you’re fuck-up is just a minor annoyance — to laugh about and tell the tales how everyone also fucked-up that one time — rather than an actual shit-storm.

jrockway 5y ago
This one is pretty tame. I feel like we have all been tempted to write ridiculous test data -- "Dear Valued Customer, we regret to inform you that the HBO Max Defraculator has been compromised and your credit card details have been sold to a self-aware purchasing bot. We are sorry and wish you the best." Sending that to your customers is a minor disaster. Sending "This template is to be used by integration tests only." will be ignored by everyone who isn't a software engineer.

deleted 5y ago

[deleted]

theshadowknows 5y ago

Automation is fun because when you make one mistake you actually make it millions of times inside of a few seconds :) been there!

werber 5y ago

I laughed when I got the email, knowing I’ve done the same damn thing. Seeing it here I’m more worried for the poor soul

pg5 5y ago

Someone I know who doesn't have and never has had an HBO account received one.

beebs93 5y ago
Yup, same here. Not sure how I was ever on their list :/

jerrysievert 5y ago

I received it. and immediately felt for the person who accidentally sent it. it happens.

jameson 5y ago

My colleague did it once. He accidentally used an incorrect API key and sent a garbage push notification like "ohai" (don't recall exactly). We were a pizza delivery startup. Few hours later, we sent another notification as an apology. "Oh Hi. Use this coupon code for 20% discount: OHAI20" Good'ol days

theoblank 5y ago

Reality is this will lose them paying subscribers, so without a doubt this is an expensive mistake regardless of who’s to blame. Large subscription businesses are very careful about when to email the segment of their subscriber base that is paying but inactive. This email just reminded a bunch of people they are paying for something but not using it, and therefore should cancel (granted, it’s unclear who it went out to). It could cause some ESP/deliverabilty problems too. On the bright side, this can serve as a reminder to us all to make sure this doesn’t happen at our companies.

Larrikin 5y ago
Everyone I've asked that is paying has said they got it but paid it no mind since it had no information that seemed relevant to them or they were a developer that thought it was funny. I'm actually hard pressed to even imagine the person that is cancelling their HBO max account because this single email was just too much for them.
- deleted 5y ago
  [deleted]
- jffry 5y ago
  I don't imagine many people will cancel HBO Max because this email was just too much, but rather because this email reminded them that they have HBO Max, and might spur them to think "I never use that so I should cancel it"

spike021 5y ago

Apparently it really was done by an "intern". >We mistakenly sent out an empty test email to a portion of our HBO Max mailing list this evening. We apologize for the inconvenience, and as the jokes pile in, yes, it was the intern. No, really. And we’re helping them through it. https://twitter.com/hbomaxhelp/status/1405712235108917249?s=12 https://twitter.com/hbomaxhelp/status/1405712235108917249?s=...

jffry 5y ago
This just gives me so many more questions. I hope that their apparently compassionate public stance translates into a healthy internal response that doesn't railroad the poor intern.
bobbyi_settv 5y ago
I guess they're going for technically correct since 100% is indeed "a portion"

whakim 5y ago

I did something very similar in my current job during my first few weeks. We had an "engineering test" script that monkeypatched the definition of "all users" (so that it returned the emails of the engineering team rather than the several hundred thousand people) before calling "send to everyone". This is obviously bad because someone might come along and change the internals of the "send to everyone" method and that person happened to be fresh-faced me. I had been tasked with making "send to everyone" a little more judicious (e.g. excluding people who hadn't opened an email in 90 days or had reported emails as spam), so I replaced the call to "all users" with some other method that calculated the recipients. And then, because I didn't totally understand how the script worked (and it seemed to be listing out some internal emails as recipients), I ran a final "engineering test" in production to see how it would look with real email content. Whoops x a few hundred thousand. I did get some pretty great responses to my apology email though.

jannes 5y ago
> excluding people who […] had reported emails as spam How do you find out whether a user reports you as spam? Wouldn’t that require some sort of feedback from Google, Microsoft, etc?
- t0mas88 5y ago
  You can get that feedback from most larger email providers. A lot of email tools can report on it automatically and make sure not to email again.
- whakim 5y ago
  In our case, Sendgrid will send a certain webhook when you attempt to send to one of these email addresses.

floatingatoll 5y ago

HBO MAX lead Andy Forssell responded: https://twitter.com/aforssell/status/1405731205274685442?s=20 https://twitter.com/aforssell/status/1405731205274685442?s=2...

pndy 5y ago

In the last August Polish internet bank mBank had two "issues": they sent out of sudden 3 notifications onto mobile devices, one even literally saying "push notification test" and the other included some of Polish diacritics [1]. Site went offline soon after these were delivered. Then few days later, database was damaged in a way some clients could see account balance and personal data of other people - that was fixed relatively fast but permanently disabled the feature of accessing recipients list in shared accounts. No idea if that was fixed - my family moved accounts elsewhere. [1] - https://niebezpiecznik.pl/wp-content/uploads/2020/08/mbank1-489x600.jpg https://niebezpiecznik.pl/wp-content/uploads/2020/08/mbank1-...

jes 5y ago

A few decades ago, I was rewriting the standard sendmail.cf that we used in the company where I worked as a sysad. I think we had about one hundred or so Sun 3/60s and 3/80s that were using it. Well, in one of my test emails, I triggered an email loop. Suddenly, all the clients were furiously mailing the mail host and queuing up additional copies as fast as they could. To make matters worse, my test mail was unprofessional in terms of the language I used. Fortunately, it was a Sunday afternoon when it happened, and the mail stayed local. But it was scary and a real pain in the neck to halt and clean up. In a future post, I’ll share how a fellow sysad and I shorted out the RG-8 coax backbone while trying to add a vampire tap. That was another kind of all night deal. :-|

iancmceachern 5y ago

I got this email!

TheBolivianNavy 5y ago

A long time ago I worked at a company that prided itself on unrestricted source code access to promote cross-group cooperation. The check-in procedure relied on an understanding of the code review and ownership policy but wasn't enforced programmatically. One day I came back from lunch to see that someone had marked an entire Perforce depot for delete. I've never scrambled so fast to try to identify this unknown user. I figured it had to be an intern and ran across the floor looking for him, only to find a poor confused intern surrounded by out of breath developers imploring him to not hit submit. Every 30s someone else ran up to try to get him to not delete the entire depot. It was the intern's first week on the job and he thought he was just deleting some local files. The event definitely inspired some access lock-downs and automated check-in procedures going forward. --- I laughed when I saw the email but was also a little pissed because I am a former, not current customer. I hope HBO Max doesn't send out an apology email. That's the equivalent of unsubscribing from an email list only to get an email confirming the unsubscription that I also have to delete.

jve 5y ago

On 30th May I received an sms from my post office at 2AM. It says that the package with some tracking number couldn't be immediatelly delivered and there will be a followup sms when I will be able to claim that package at the local post office. I asked at the post office about it and they said: "oh, many people ask about it. We received that sms too. That is to be ignored." I wonder why they couldn't send followup sms and save people trip/call to local post office.

notimetorelax 5y ago
I’m assuming folks that responded to your call are not the same folks who can send the follow up SMS. Happens all too frequently when e.g. developers are detached from production and oncallers are dying under the pager load.
- jve 5y ago
  Well, those employees should pass that info UP the chain and let them know that this mistake causes inconvenience and confusion and they should clean up the mess. Ofcourse that lady is not the one sending sms... I wonder if they are testing new functionality - I'v read that we will be able to have our own pickup station right at my house, for some subscription price or something like that. So packages/postage will come delivered directly to my door behind a closed box (given it is being delivered by national post office).

wyldfire 5y ago

Aside: I'm a big fan of HBO's content but their app is really poor quality. Playback frequently is unable to start or halts midstream. I've noticed this on distinct devices (android, AppleTV). DRM run amok? I don't know but you would think making playback work consistently would be near the top of their list. It's gotten slightly better over the last year or so. But it still happens way more often than it should.

hyperman1 5y ago

I heard how a company in the long past got a phone call from an elder customer: I've got such a strange letter from you. Reception asks what it says. Says the customer: I don't know yet, they're still busy putting it in the hallway. Huh? It turns out an error was made in a mass mailing, and every letter was sent with the first address on the list. The list was roughly sorted by birth date, so the eldest customer got all of them. Postal office workers drove with a van to the customer, and duly delivered bags and bags of mail to the same address.

century19 5y ago
Same thing happened somewhere I worked, except all mail was being sent to the company HQ. It was loaded on lorries but someone at the sorting centre had the foresight to phone up and ask "is this correct"?

theshrike79 5y ago

Bah, amateurs. I did this with physical mail. I was developing an e-letter integration. Basically I had to push a specifically formatted text file to an FTP server, from where the Post office's systems would grab it, print it and mail the physical letter. There was a specific "this is a test" bit that had to be set so that the actual letter wouldn't get sent. So... I kinda forgot that bit at one point and received a hefty pile of letters addressed to me to the office.

dataviz1000 5y ago

The email is innocuous. I opened the headers to see what was going on and why it got past spam blockers. My next thought was how much more valuable this mistake is than most their advertising budget to get attention.

deleted 5y ago
[deleted]
deleted 5y ago
[deleted]

herunan 5y ago

I work at WarnerMedia, who own HBO Max, and I thought this was a test using an employee account pool. Never mind, then!

aequitas 5y ago

This is why you always use the special use[0] domain names like example.com for your tests or documentation. [0] https://en.wikipedia.org/wiki/Special-use_domain_name https://en.wikipedia.org/wiki/Special-use_domain_name

makach 5y ago

These things happens. It's not your fault but a consequence of things that can improve. You'll recover. Misconfiguration is also cyber securiy threat #1 these days.

deleted 5y ago

[deleted]

callamdelaney 5y ago

It was the intern! Oops. Source: https://twitter.com/HBOMaxHelp/status/1405712235108917249 https://twitter.com/HBOMaxHelp/status/1405712235108917249

starfox64_ 5y ago

This reminds me of when I took over a project after a developer that left they company. They had just added SMS notifications to production but to my horror left some test code that would text you "Hey motherf**r" every time you would change your phone number. To this day I still wonder how many people got that text.

pachico 5y ago

I bet what happened was that their integration environment has unredacted production emails.

bowlingx 5y ago

Yap, been there :/.

beshur 5y ago

Well, this is a true integration test, when you finally reach real users :)

treyfitty 5y ago

Apparently, it was by an intern https://twitter.com/HBOMaxHelp/status/1405712235108917249?s=20 https://twitter.com/HBOMaxHelp/status/1405712235108917249?s=...

DangerousPie 5y ago

Interesting that it appears to have gone out to former subscribers too. They really shouldn't be retaining those email addresses, should they?

lm28469 5y ago
Are you protected by GDPR ? If so yes, if not it's just business as usual. Gather as much data and keep them just in case you need them in an hypothetical future for an hypothetical use case

jakub_g 5y ago

patio11 shared on Twitter how he pushed a faulty code to prod *just before changing apartments* so that it started sending email, sms and calling hundreds of people every 5 min for 13 hours: https://twitter.com/patio11/status/1405704339969220615 https://twitter.com/patio11/status/1405704339969220615 https://twitter.com/patio11/status/1405706868110987269 https://twitter.com/patio11/status/1405706868110987269 https://twitter.com/patio11/status/1405707285268103169 https://twitter.com/patio11/status/1405707285268103169

rojcyk 5y ago

When I was working at Kiwi.com as a senior product designer one of the guys on the development team did something very similar. We were testing a new push notification refactoring and managed to send a push notification "Test test" to our whole Android user base (couple of hundred thousands of users). Interestingly enough, it was a great reactivation campaign. We bumped bookings by a noticeable margin, and since we redesigned our "rate us" flow couple of weeks before, we had a huge influx of 5* ratings as well.

sackerhews 5y ago

Stuff like this can unfortunately happen. This is a reminder not to use use obscenities or jokes in your test data. Remember Mr. Rich Bastard: https://www.snopes.com/fact-check/dear-rich-bastard/ https://www.snopes.com/fact-check/dear-rich-bastard/

ElViajero 5y ago

> "This template is used by integration test only." No, it is not only for integration test. It is also used in production. They need to update that description. :)

napolux 5y ago

We once sent 1 single user 100k emails. We sent him flowers to apologize :P

diveanon 5y ago

Pretty sure this will happen to every web dev at some point in their career. It's why I strictly enforce professional test messages and test users, nothing worse that doing a product demo with users named Donald Trump and Osama Bin Laden. Hope the guy didn't get fired, things like this happen because of multiple failures in dev ops, not just 1 guy pressing the wrong button.

tottenhm 5y ago

I can’t wait to watch Integration Test Email #2 at the same time next week.

fleddr 5y ago

Amateur mistake, compared to my professional mistakes. I once wrote a bot that sent email within the company (100K+ employees). I kick-started the bot on a server remotely and only then discovered it's an endless loop. It required server admin rights to stop it, which I did not have. I couldn't immediately reach server admins so had to physically drive there. An hour or so later somebody helped me kill the process. The emails already sent could not be cleared out server-side, which meant that recipients had freezing email clients for over a week, unable to handle the volume, a typical 300K new emails per recipient. They had to Ctrl+A and Delete 100 emails, do this for the next 100, and so on, whilst not deleting real and useful emails. I pretty much destroyed email for those people. I don't just destroy things at scale though, also at home. Around the time we had our first home broadband internet connection, I set up a web server and just kept my PC running. Unknown to me, the web server software included an email server with open relay enabled by default. About 3 days later, my dad complained about the internet connection not working. The IPS detected the issue (millions of emails sent out via my home server) and gave us a red card, fully shutting us down, permanently.

colejohnson66 5y ago
To avoid accidentally deleting useful emails while “Ctrl-A-ing”, could they have run a search by sender or some other criteria first? Then only delete the search results.
- fleddr 5y ago
  In the email client? Probably yes, if they know how. This was a very long time ago, early 2000s, and the email client was the dreaded Lotus Notes software.
  - jimmaswell 5y ago
    I had to use that client recently. I can't believe it doesn't even have a proper email search function. Best you can get is ctrl-f on the open page or sorting all your email alphabetically.
5y ago

nofinator 5y ago

For about 24 hours last summer, Citizen app on Android would play the audio from a "Who's that Pokemon? Rick Roll" video when there was an alert. They quietly fixed it. I can only imagine that it went to prod because it was a goofy test alert sound that accidentally got promoted out of a QA environment?

joren- 5y ago

To share my mistakes as well: After launching a web app we noticed severe performance issues and had to roll back to a previous version. To identify the performance problems introduced in the new version, I wrote a script mimicking a thousands of typical user interactions. This also included sending email. In the moment, we forgot to properly configure the dev environment and the emails went out.. but it did not stop there. This is a good decade ago and some clients still preferred to get faxes instead of emails. We used a service which automatically converted emails to fax messages. We tried to intervene but were not able to stop thousands of fax messages and basically DDOS'd many fax machines and then had to call the affected clients..

adius 5y ago
Maybe this prompted some people to switch to email? Thanks for your service in that case! =P

mcintyre1994 5y ago

Since we're sharing email sending mistakes: We wanted to start sending users occasional summary emails about activity on the platform that they might be interested in. We wrote the email generation logic first, and then the plan was to start adding different activity events and dogfooding the emails internally to see how they looked in practice. This would be fine because while we developed only internal users would have activity data, so only we'd get emailed. But there was a bug in the "is this email empty" logic so we once sent every user an email with the subject of something like "Check out your recent activity on X" with a few headings and absolutely no activity content. Oops.

SomeHacker44 5y ago

This is a perfect mistake. Embarrassing but mostly harmless. Probably not too expensive (depends on ESP cost). Great teachable moment. I wish I could get my engineers to make more mistakes like this. :)

detaro 5y ago

And this is why you should be careful with "funny" test messages/placeholder names/dummy products etc. This could be a lot more trouble if it weren't obviously a technical thing, although a clear sentence along the lines of "if you are a customer and received this email, sorry, please ignore it" would maybe be a slight improvement.

yaur 5y ago

Not much to add to this, except that I also got this email sent to an address which I have only given out to Hulu.

jedisct1 5y ago

Intern or not, everybody makes mistakes. Are we talking about heart surgery here? No, we're just talking about an insignificant test email. Nobody was hurt, and it reactions were actually fun to read. So, whoever unintentionally did that: this is okay. And if your manager says it's not, this is not okay.

at_a_remove 5y ago

I got this email, shrugged, deleted it, moved on. No harm no foul.

notacoward 5y ago

As is usually the case with such things, this wasn't really the intern's fault. The question I'd be asking is: why wasn't there proper access control in place to stop this? The system actually sending the emails should not have allowed "testuser" to send to a production list. So, second question: when this was proposed during some planning meeting (and it almost certainly was), why was it shot down? That's the process problem right there.

lh15 5y ago

I once wrote an endless loop triggering thousands of Twilio SMS messages. Took a few minutes to kill the server