10 ms·

Aww I’m feeling bad for the poor engineer who is saying “crap” a thousand times now, or is blissfully unaware that their day is about to get much worse. Once d

• chrischen 5y ago
  Honestly I'd rather them not send an apology email... because then I'd have two useless emails. And in the grand scheme of things, it's literally just one more email I receive during the day.
  • cheschire 5y ago
    The first email was useful to someone! An apology email is useless to everyone.
    • diegoperini 5y ago
      I disagree, it's definitely useful. It's useless only if it feels/is dishonest. One way to avoid sounding dishonest is letting the dev team write the small apology piece.
  • smelendez 5y ago
    I think in the case the first email is too cryptic not to follow up without being unsettling to people.
    • acomjean 5y ago
      I got it. It one line and includes the word test so I just ignored it. It wasn’t scary but it wasn’t clear what it was.
  • chrsig 5y ago
    I for one am happy to know that they're running integration tests at all, so I don't think they have anything to apologize for. It was just an email -- I get many uninvited emails per day. This uninvited email happened to be from an engineer rather than a marketer.
    • ljm 5y ago
      The first thought I had, though, was if some team decided to pull down their prod DB for testing so they had 'realistic data'. Because that would be a much bigger problem than sending an email by accident.
  • pestaa 5y ago
    Of course you say sorry after you bump into someone. It's basic courtesy.
    • Grustaf 5y ago
      Yeah but you don't call up some stranger on the phone to say sorry for pocket calling them.
      • pestaa 5y ago
        These were most likely paying customers on the quick dial.
        • corobo 5y ago
          I can't get HBO max here (UK) and I got the email. I don't recall signing up for anything. If I get a moment I might GDPR the info out of them but honestly loads of spammers have my email it's nbd. e: Going by past emails someone decided to change their account's email to mine (which HBO was fine with, no confirmation required, hope the user can still use their account).. Don't you just love end users.
          • pestaa 5y ago
            Be glad! Sounds like you could have a subscription for free.
          • machinerychorus 5y ago
            Sounds like it could be the scam where they put in your email address and hope that you pay the bill for them: https://jameshfisher.com/2018/04/07/the-dots-do-matter-how-to-scam-a-gmail-user/ https://jameshfisher.com/2018/04/07/the-dots-do-matter-how-t...
            • corobo 5y ago
              Ooh I'd not come across that one. Cheeky bastards (if so)
  • yummybear 5y ago
    However - most of HBO's customers are non-technical and will be wondering what kind of test this is and if they should be worried.
  • jamesdmiller 5y ago
    HBO should email everyone to give us all the choice of whether to get an apology email.
  • samus 5y ago
    It depends what the email was about. If its content reads like "test email" or some nonsense like that, nevermind. But if it looks like a legit email that would have significant consequences for the recipient, were it legit, it should definitely be clarified what's up. Also, a well-crafted email will hopefully prevent people invoking the GDPR on the sender.
    • commandlinefan 5y ago
      I got one. It says: Integration Test Email #1 This template is used by integration tests only.
  • commandlinefan 5y ago
    Well, I got my integration test e-mail yesterday but still no apology e-mail, so I think they listened to you.
• deleted 5y ago
  [deleted]
• thelopa 5y ago
  I had a experience like this, once. Luckily it was less visible, but I felt like a fool all the same. I came out as trans and changed my name last year, and with the name change I set up a new email alias for work. Then I set up automation to send out a gentle reminder email about the change for people who emailed my old alias. It worked fantastically for a few months… right up until the point that (due to a series of individually innocent events) the automation ended up running across the entirety of my 7 years worth of inbox. Everyone who had mailed me over the 7 years prior to the name change started getting the reminder email. One reminder for each email they had sent me. The worst part is that due to a bug in the email automation stuff, emails sent by the automation weren’t preserved in my sent box. So I don’t even know how many people I spammed. If I had to guess, I sent dozens of emails to the CEO and other execs, hundreds to my director, and thousands to people who worked closely with me over the years. I learned a valuable lesson that day.
  • squeegmeister 5y ago
    Ouch
  • snarkypixel 5y ago
    What was the valuable lesson? :p
    • throwawayboise 5y ago
      Don't write your own email auto-responders?
      • thelopa 5y ago
        You joke, but there is a profound truth to this. Don’t reinvent the wheel. To be fair, it’s not like I wrote any real code for my auto-responder. I used a slightly janky mail tool that could, if held just right, be used to set up an auto-responder. I should have looked for something a bit more bulletproof and focused, but I wanted to play with this specific tool and see what I could make it do. Mostly, I’m kicking myself for not thinking to put safeguards in. Dependencies can fail in unexpected ways, and I should have set up my auto-responder to be a bit more defensive.
• userbinator 5y ago
  Once during a system upgrade we ran some scripts and they triggered emails to about 8,000 people before we realized it (would’ve been 150k people otherwise). The next day was all about clean up, sending an apology email etc. On the bright side, if you can accidentally send an automated email to that many people, then sending another email to them to apologise is unlikely to be a manual effort either.
  • ryukafalz 5y ago
    When you've interrupted the process partway through as they did, figuring out which part of the list you need to apologize to may well be significantly more effort.
    • edgyquant 5y ago
      This is why at my startup we have a “sent_emails” table with the to and from addresses, email type and a template id (if applicable.) Saved a lot of headaches
      • mschuster91 5y ago
        Additionally, you have some sort of paper trail you can use in GDPR or T&C disputes.
      • atonse 5y ago
        Yup and we have deduplication logic anyway from day one. It was easy for us to figure out the 8,000 since we log every message.
    • generalizations 5y ago
      Just restart the script and send it to all 150k people, then just send the apology to the whole list.
      • notwedtm 5y ago
        Found the DevOps
• scrame 5y ago
  Can relate. Had an excitable junior engineer working on migrating between platforms, spammed the global list with '123456'
  • dividedbyzero 5y ago
    Without knowing any details, that you're framing it this way (their fault for being junior and "excitable") and that you don't mention any failed safeguards and process failures (implying that their mistake was an easy one to make) doesn't let you (and that org) appear in a great light. This sort of thing is a huge org smell for me; people will always make mistakes, how you deal with that fact tells a lot about the quality of leadership. And what exactly is the "excitable" bit supposed to tell us?
    • lal 5y ago
      It's probably supposed to tell you this is a humorous anecdote about a mistake someone made and not a thinkpiece about all of the valuable lessons this company learned.
• piyh 5y ago
  An HR person was poking around the admin panels of a payroll system and accidentally clicked a notify on SSN change box. An email was sent with the before and after social security number went out to 20,000 people.
  • eru 5y ago
    Seems like a bug in the system?
• teeray 5y ago
  That engineer should absolutely point out that despite the mistake, he just created the most engaging newsletter email ever in defiance of modern marketing practices. The real cosmic brain strategy is to run with this somehow and turn it into something like those guerrilla marketing campaigns.
  • chromaton 5y ago
    It's a promo for the new season of Silicon Valley.
  • dawnerd 5y ago
    It’s number two trending so it sure worked!
  • pixiemaster 5y ago
    that’s correct. happened to me as well, we send out newsletter to a b2b ecom site with all links to staging, behind htacess of course: so no tracking pixels, no images etc. some complaints, lots of „hey you did something wrong - even more of „i can’t open it please send again“ — and it was the best week in terms of sales ever.
  • bjacokes 5y ago
    The Accidental Integration Test on Purpose. Larry David would be proud.
  • peterkos 5y ago
    That’s honestly what I thought it was at first!
• whalesalad 5y ago
  Badge of honor to be honest. Even if HBO Max fired me I would forever be stoked to be "that guy/gal" who sent out an integ test message to the entire userbase.
  • corobo 5y ago
    If HBO fired them for this then they'd have to bring in someone without the "Don't accidentally send a million emails" training to replace them Intern isn't going to be sending mass email again without a double check
• flukus 5y ago
  > Aww I’m feeling bad for the poor engineer who is saying “crap” a thousand times now, or is blissfully unaware that their day is about to get much worse. I hope not. It sounds like the test database was not being anonymized but sometimes things like this can be as simple as not selecting a debug build in Visual Studio, either of which is an organizational issue and not an individual one so he shouldn't be punished. This is especially true if you have corporate buzzwords like "taking ownership and responsibility". No one will take responsibility if there are punishments for owning up to and admitting mistakes. Odds are they feel pretty bad about it already. While we're sharing personal anecdotes. Parents get very upset when they incorrectly receive truancy reports because you forgot to check the IsDeceased flag...
  • onion2k 5y ago
    I hope not. It sounds like the test database was not being anonymized Taking a copy of a production database and using it for tests is a bad idea, even if you believe you're expunging any private user data. Development, staging, and test environments just shouldn't ever have access to production data. If you're at a company that's ISO27001 certified for data security it even goes as far as most employees not having any access to data. I've never seen any production data for the app I work on. https://en.m.wikipedia.org/wiki/ISO/IEC_27001 https://en.m.wikipedia.org/wiki/ISO/IEC_27001
    • gls2ro 5y ago
      I agree about the part of not accessing information from production. But I am wondering how could we debug or test something which happens only on production? I ask this because there are some bugs that can appear at the intersection of code and data. So far my strategy is to do the following: 1. Only one person can access production DB. This person will do a backup copy and encrypt it to an internal storage. 2. Another one will get the backup and run an anonomizer script on data. The anonimzer is still up to debate what it should do after the obvious cleaning of personal data from user accounts. One important (and hard step) is regenerating the uuids but keeping foreign keys integrity. At the end this person will create a new DB internally with the anonimizer data. 3. Someome reviews the new DB and marks it as ready to be used Then a dev can ask access to this fresh copy. In some teams I played with making this process full automated until review. But then if there are bugs suddenly we have a live internal DB with customer data which is not wanted. As an alternative but only for small projects I wrote once a script which analysis the DB data and tries to create fro, scratch a similar data structure but with fake data.
• acituan 5y ago
  I sympathize because when I was a junior engineer once I accidentally emptied the test email spool sending the entire company including upper management all sorts of fake test emails accumulated in the past years about hires and fires and whatnot. What is more, a coworker convinced an HR person to play a prank and call me to HR office next day for giggles. Needless to say it was stressful and infuriating at the same time. Now I know better and if somehow they are reading here, I would advise the person to just chill and don't take any shit than necessary. If you were not one of the few people with root access but somehow still had the capabilities for mass emailing in prod, that is not your problem, it is an organizational problem. For an operation at the size of HBO, anything prod has to be behind sufficient failsafes and a peer reviewed process (except maybe for a very rare "break glass" emergency). Hope there will be a good, rational postmortem that can cool headedly identify the root causes and create action items for the actual stakeholders. If your shop is worth its salt, there wouldn't be performance evaluation consequences for you. If there is, no worries either, it is time to look for a better place.
  • saghm 5y ago
    > What is more, a coworker convinced an HR person to play a prank and call me to HR office next day for giggles. That's the type of thing HR people should be putting a stop to, not literally being a party to. I don't have any illusions about HR being there for the employee rather than the employer, but I can't imagine working for a place where HR is abusing their authority to add stress and shame solely for their own amusement.
• cortesoft 5y ago
  Similar story... about 10 years ago, I had written a really simple script to email all our customers. It worked great for a long time, but then suddenly we went over 1000 customers. My script was supposed to try to grab batches of 1000 customers and keep looping until it ran out of customers (signaled by having retrieved less than 1000 customers in my last request for the next batch of 1000 customers). My script was missing the offset part of the query, so after we hit 1000 users, it just kept looping, sending the same email over and over to our first 1000 users. I felt so bad that day. From then on, sending out emails was this whole huge process that involved queuing them all and then having like 6 people review to make sure we didn't mess it up.
  • wiredfool 5y ago
    When I worked on email systems, my worst nightmare was a Sorcerers Apprentice sort of problem.
  • prox 5y ago
    I always was super careful with that stuff. The final send function wasnt unlocked until knew for certain all worked as intended.
  • yawnxyz 5y ago
    oh man I just wrote a piece of code that lets me write any markdown, push a button and it just sends it out to our ~1-2000 users. An hour later I just commented it all out, and wrote a note to myself: "if you need this, uncomment and push back to stage". Just having that code even sitting around makes me nervous
  • samus 5y ago
    We introduced an allowlist on all our testing and staging environments to ensure that only certain recipients can get email. We also make sure that no email address in these databases would work, unless we really want to send to it.
    • atonse 5y ago
      We did the same thing after that incident. It’s worked beautifully.
• Semaphor 5y ago
  I had a bug in our invoice sending script. Someone had been getting thousands of copies of their invoice. They called and politely asked if we could stop sending them that mail :D
• nathias 5y ago
  who thought sending another spam mail as an apology for sending spam mail was a good idea?
• corobo 5y ago
  I appreciate the solidarity coming in from the replies to their response[1]. We've all done it. If you haven't broke production you haven't broke production yet. https://twitter.com/HBOMaxHelp/status/1405712235108917249 https://twitter.com/HBOMaxHelp/status/1405712235108917249
  • deleted 5y ago
    [deleted]
• sodafountan 5y ago
  The bigger and more concerning issue would be recipients complaining about the incoming email and marking it as spam, which could damage the reputation of HBO Max's domain and potentially send future legitimate messages to spam. (provided the integration tests don't use a non-primary domain for sending) I learned this the hard way in my last role, I worked for a small company that wanted to do custom email marketing. I was pretty gung-ho about it, I thought I'd just set up a script to loop through contacts and use mailgun to send the email from a custom domain. As we used the tool we saw a steady drop-off in click-throughs to the site, the majority of our messages were getting caught up in spam filters. Turns out there's a whole science to email marketing, how emails should be structured and formatted etc. A lot of times the criteria for spam is how often the domain was flagged for spam in the past, the length of the email, the contents etc. Ended up relying on this tool quite a bit: https://www.mailgun.com/deliverability/email-spam-checker/ https://www.mailgun.com/deliverability/email-spam-checker/