10 ms·

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Glo

• kazinator 5y ago
  What? The most inexusable thing when engineering mistakes happen is something like: failing to bring back people's dead friends and family members. Not that some customers of some groupware application weren't communicated with for a week.
• iamtheworstdev 5y ago
  Can confirm. Saw them there while I was on vacation.
  • benreesman 5y ago
    Jesus, if there was ever an example of the internet making the world smaller. When do execs living it up at the fucking Wynn Encore while the house burns down start to not get another job? They’ll keep pulling this shit until it cost money.
    • benreesman 5y ago
      For clarity: I went through a period where some combination of self-indulgence and legitimate life crisis caused me to take my eye off the ball when it mattered. I’m still trying to kickstart a second act years later, because I’m trailer trash and it’s hard work when you’re that.
      • gjvc 5y ago
        Everyone deserves a second chance. You'll be on top again before you know it, and when you look back it'll have been easier than it seems now. Fair wind and God speed!
  • broknbottle 5y ago
    Thanks narc, now our proper holiday to Malia Greece was put on indefinite hold. Vegas was only supposed to be the pre game warm up.
• syshum 5y ago
  sales never takes the blame. If anyone is fired it will be scapegoats in engineering once they have busted their ass to restore their reward will be the door
  • systemvoltage 5y ago
    This is an engineering problem. They should own it and improve things, make sure it doesn't happen again. Also, GP's quote > Engineering mistakes happen. I don't like this statement because it offers consolation at the expense of unintentional normalization.
    • buscoquadnary 5y ago
      And coders that say all code has bugs are just defeatists that are trying to make excuses for being lazy. Sometimes manure will always hit the fan. Being robust means being able to handle that.
      • systemvoltage 5y ago
        A culture where mistakes are taken too seriously or too lightly leads to problems. Also it depends on what stage of the product cycle (Innovation/Rapid Development vs. Robustness/Quality). I'd argue that Atlassian products should err towards robustness and high quality. Not trying to break any new ground.
      • jacksnipe 5y ago
        I think this is obviously incorrect. Human error is probabilistic, and the probability of making an error cannot be zero. On the flip side, it’s infeasible to use only provably correct systems; not lazy, but literally not a practical option due to compute costs, developer time, what formal techniques can even be applied to the problem at hand, etc…
        • tinco 5y ago
          It's not about making the probability of an error zero, it's about making sure you can recover from every type of error effectively. They've underbudgeted for engineering, and they're feeling that now.
• naoqj 5y ago
• madmulita 5y ago
  They claim they test backups quarterly yet they don't have a procedure in place to restore the operation. We all know your backup is not tested until you restored everything successfully. This is not an engineering mistake, it is a flat out lie.
  • iancarroll 5y ago
    Well, their explanation makes sense. These are multi-tenant environments where not every tenant was affected; sensibly, the backups appear divided by environment, not tenant. You can’t blindly revert to an environment’s last backup in this scenario, although you’d think they would have done it before.
    • theteapot 5y ago
      Not having per tenant backups is sensible? Seems like a bit of an oversight. Doesn't really matter if the hosting is multi-tenant or not.
      • iancarroll 5y ago
        You can imagine problems restoring one individual tenant's data to an otherwise active database with many tenants; i.e. any cross-tenant primary keys which will have shifted since one tenant's last backup. Separating the backups wouldn't help with the restoration.
        • sverhagen 5y ago
          Why not restore the tenants to a different environment that is not otherwise active? At Atlassian's scale you would expect them 1) not to be running all the things on one server anyway, 2) have some existing ability to move tenants between environments for legal or performance reasons, 3) have the ability to backup/restore single tenants, and so on. I am not arguing that the predicament they are in now is real, for them, now. But I have worked at much smaller outlets where this worked fine, not because they were smaller, but because they had that particular shit in order, which is entirely a matter of priorities. Nothing, and I mean absolutely nothing, that Atlassian has to offer is rocket surgery-kind of hard... yet, here we are... not being particularly surprised at all.
• _sword 5y ago
  Fair criticisms on response times but regarding Vegas, it was their annual user conference last week in Vegas.
• julesallen 5y ago
  No argument on the crappy comms. If I was in customer success at an enterprise vendor I doubt I'd be let anywhere near the tools to get this back up and running. These guys are generally in the way rather than helping in a situation like this. Head of engineering or some product rather than customer support? That might be a different outcome.
• polote 5y ago
  There is not much csm can do when an outage like that happens more than waiting. So I don't see why partying at the same time could be an issue. Thats the engineering which made the mistake. So even though they could have communicated better ( and we don't have all the details, we don't know). The true people at fault are Engineering and product leaders. Engineering mistakes is not an excuse to blame others, and there is difference between a mistake and removing data of production customers
  • deleted 5y ago
    [deleted]
  • cerved 5y ago
    The number one priority of someone maintaining business should be ensuring communication if you want to keep business
• opensrcken 5y ago
  Unless you know, with certainty, who is responsible for customer-facing communications , how can you publicly call on a specific person to be fired? Even if you did know, and are correct in your implication, how do you know that she didn't want to do the right thing but was overruled by people above her in the chain? Overall, this is a pretty irresponsible comment, and reflects the kind of mob mentality that I would expect in Reddit, but not HN. And the bit about Atlassian employees being in Vegas has nothing to do with anything - as if the entire company is supposed to shut down its planned celebration because of an incident that a small subset of the company should be handling.
  • wly_cdgr 5y ago
    They are literally the head of "customer success". The buck should probably not stop ONLY with them, but obviously they're one of the people it needs to stop with
    • mdoms 5y ago
      You seem to be implying that customer success is a customer support role. It isn't. Customer success is about helping customers get the greatest business value from your product. They are not going to be munging databases and wrangling backups. Support engineers (among others) do that work.
      • wly_cdgr 5y ago
      • throwaway1777 5y ago
        Well right now customers are getting zero value. It’s a bad look to be partying during an all hands on deck emergency.
        • carlmr 5y ago
          More hands on that deck probably don't help.
• mrlinx 5y ago
  She replied to that in the comments of that post: > Yes Andy it was and I didn't realize that my scheduled posts were still going out. They have now stopped.
• mdoms 5y ago
  A lot of people talking about "firing" in this thread. That's not how Atlassian operates, to their credit. It probably comes from their Australian DNA.
  • takeitaussie 5y ago
    True. Like their laid back approach to operations.
• _moof 5y ago
  These are base instincts speaking. Emotional, vengeful, a desire to punish because you've been hurt. It's perfectly natural to feel that way, but it's not reasonable, fair, or effective.
  • zaphirplane 5y ago
    I don’t think calling for the sacking of people is a good thing However it doesn’t imply there is vindictive drive some people will not have a good month career wise, some people will lose trust, some people will be an consequences of regaining the public confidence
• PragmaticPulp 5y ago
  > Actually reading a bit more, it seems like their customer team was partying in Las Vegas instead of taking care of business: https://www.linkedin.com/mwlite/feed/hashtag/atlassianteam22 https://www.linkedin.com/mwlite/feed/hashtag/atlassianteam22 This was a conference they hosted, not just some Atlassian team members partying in Vegas: https://events.atlassian.com/team22 https://events.atlassian.com/team22 Thousands of Atlassian customers bought tickets, flights, and hotels for the event. It's unreasonable to suggest that they should cancel it all days before the event. > Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 She responded that it was a scheduled post. This is normal on LinkedIn: Marketers write a lot of posts all at once and schedule them to come out periodically. They're not literally sitting down in front of LinkedIn and typing out their daily puff piece. Please, let's drop the pitchforks. It's not a good look for us on HN, especially when the facts are being completely misinterpreted.
  • staticautomatic 5y ago
    Fair enough, but anyone who’s been around the block on crisis communications knows that you’re supposed to hit the big red “stop all regular communications until we get a fucking handle on this” button.
    • Sebguer 5y ago
      You think Atlassian should have the ability to stop posts on all of their individual employees' social media posts?
      • jupp0r 5y ago
        It’s quite common that employees are only allowed to share specific information about companies they work for on social media. Atlassian most likely has that power already.
      • deleted 5y ago
        [deleted]
• avl999 5y ago
  People who call for other people's firings in organizations that they have no visibility into are so weird. This post reads like a tech outage's version of cancel culture where trying to find someone to blame and skewer for an injustice is more important than actually determining how much (if any) blame they deserve for it Also posting a LinkedIn event photos with people's real names and pictures in a top post on HN along with provocative framing like "partying in Vegas" about this outage is pretty shitty. Atlassian is a big company and has a huge engineering and sales department, you have no idea if any of those people at the Vegas event had anything to contribute with the outage response. For example in your link there are literally people talking about some new mobile app their team is launching at the event, I doubt any of those people are involved in the outage response. No one posting on HN unless they work at Atlassian in a leadership role is in any position to even start assigning blame, call for firings or publicly shaming people (the later of which you shouldn't be doing even if you 100% knew for a fact that they were at fault). Do better.
  • rootsudo 5y ago
    I agree. Customer success is a support role, this was an engineering mistake. Can't blame support for something an engineer did.
    • femiagbabiaka 5y ago
      Seems like you're still assigning blame. Incidents are rarely if ever monocausal. The fantastic and accurate point the GP made is that fingerpointing is pointless. Much better to seek to learn and understand, which is always difficult but definitely can't be done from the sidelines without speaking to those involved.
      • rootsudo 5y ago
        Which is why I used mistake, the context is blaming support for an engineering action. Support is purely reactive.
    • roywashere 5y ago
• deleted 5y ago
  [deleted]
• dewey 5y ago
  To be fair that was a scheduled post and she took care of it already: > Yes Andy it was and I didn't realize that my scheduled posts were still going out. They have now stopped.
• sirrocco 5y ago
• Ecstatify 5y ago
  Apparently it was a scheduled post, it looks like corporate bs so I highly doubt she wrote it. https://www.linkedin.com/feed/update/urn:li:activity:6918235769572577280?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6918235769572577280%2C6920135404423675905%29&replyUrn=urn%3Ali%3Acomment%3A%28activity%3A6918235769572577280%2C6920153567341228032%29 https://www.linkedin.com/feed/update/urn:li:activity:6918235...
• sjs382 5y ago
  A lot of the replies to this thread seem to be running with their own assumptions & definitions of what a "Customer Success" dept/org is. I've personally never worked for a company that used that language, so I went looking and found the job description for this person's role: https://startup.jobs/global-head-of-customer-success-remote-atlassian-1843130 https://startup.jobs/global-head-of-customer-success-remote-... I'm still unclear where "Global Head of" for something like this fits in an org chart (Who do they report to? Who reports to them? Is it within a marketing, sales or customer support tree? Etc). Title inflation and whatnot considered...
• druud62 5y ago
  When you depend on the clown, it will hit you some day. So if you are hit by such, now finally move away again from only depending on the clown, and (for example) re-value your on-prem-ity.