10 ms·

Update on the Atlassian outage affecting some customers

132 points by taspeotis 5y ago

PicklesMcPuffin 5y ago
“End-to-end, it takes between 4 and 5 elapsed days to hand a site back to a customer.” Cool marketing strategy. Maybe you’ll get your data in a business week.
aunty_helen 5y ago
After listening to in-person talks from Atlassian in the past, I got the impression of them having a smugness from being a big fish in a small pond (Australian tech.) Their original and early Jira product was great and put them out in front, but I couldn’t name any other product of theirs except Bitbucket which is very much a second place competitor in their market. Jira itself has faded and been caught up with too. Reading through the post and seeing the level of support some people on Twitter have been tweeting about, I would take this as a great time to look at other offerings out there.
- jeffybefffy519 5y ago
  I think Confluence is a genuinely good product. Its nice to see that even Atlassian struggle with day to day sysadmin of Atlassian tools
  - karmakaze 5y ago
    A lot of people look down on Jira. My experience was that Confluence was much worse. It turns writing a couple technical notes into a publishing journey with missteps and failures along the way.
    - VTimofeenko 5y ago
      For me Confluence dramatically improved in experience when I learned that it's possible to write files locally using $FAVORITE_EDITOR in confluencewiki format (or markdown if you don't care about macros) and post them through API.
      - clhodapp 5y ago
        Can you share anything more about your technique? As far as I know, Confluence hasn't supported a markup format natively for several years. I've found some tricks to allow me to author stuff in Markdown but it seems to come with a lot of limitations which ultimately mean that I have to start using the web editor as soon as I want to start collaborating with others.
        DocTomoe 5y ago
        In Confluence Editor, under the Macro "Plus" button, you should find an entry called "Markup" pretty far up the list, which accepts "Confluence Wiki" Markup (which is just Markdown + Macros). This works for both on-premise as well as cloud instances.
        VTimofeenko 5y ago
        I wrote a tool that automates the posting part[1]. Unfortunately I could not solve the collaboration problem either. Setting the body of the page overwrites it. I did see some examples of editing HTML through similar approach but never tried it myself. [1]: https://github.com/VTimofeenko/confluence_poster https://github.com/VTimofeenko/confluence_poster
  - RexM 5y ago
    I read somewhere recently that confluence is write-only because search is so bad nobody will ever find what your write. I haven’t used it in years, but that seems accurate. I could never find anything unless I knew exactly how it was worded in what I’m looking for.
    - djbusby 5y ago
      Putting markdown text into SQLite with FTS5 works amazingly well for search. How come they couldn't just bolt that in?
    - wink 5y ago
      Yes, it's actually so bad that you wonder what part of the picture we're missing.
    - xen0 5y ago
      My experience with Confluence's search was that it's a mistake the press Enter. That is, the pop-up results that appear as I typed where significantly better than the actual "search results" page.
      - junon 5y ago
        I also vaguely remember this being the case. Confluence always felt so bloated to me.
  - philliphaydon 5y ago
    When they first acquired confluence it was great. Now I can’t stand it. Is so slow now. So clunky.
- jay_kyburz 5y ago
  I had Jira inflicted on me before Atlassian was big and Jira was terrible back then too. I always liked FogBugs.
  - macspoofing 5y ago
    >I always liked FogBugs You lost me there. In no way was FogBugz a better product. It was at best mediocre by 2010.
    - LilBytes 5y ago
      Current user of FogBugz, and prior user of JIRA. Jira is much better. We also use SharePoint and Confluence is so much better. Honestly I get the criticism of both JIRA and Confluence but till you've used the other big names in that space you don't realise how bad it can get.
      - jay_kyburz 5y ago
        Fogbugz was on the downhill as early as 2010. I always assumed that when Fogcreek started working on Trello they pulled the A team off Fogbugz.
      - macspoofing 5y ago
        >Current user of FogBugz I'm so sorry. Fogbugz has been in maintenance mode for years now, stuck in the middle of a half-finished redesign.
- synicalx 5y ago
  Having dealt with the pre and post sales support quite extensively over the years, I always got the impression they were a big fish in way over it's head, and with a tail that didn't work so good.
- lbotos 5y ago
  Bitbucket was also an acquisition (Most of their offerings were acquisitions). https://techcrunch.com/2010/09/29/atlassian-buys-mercurial-project-hosting-site-bitbucket/ https://techcrunch.com/2010/09/29/atlassian-buys-mercurial-p...
- zeckalpha 5y ago
  Trello? OpsGenie? Statuspage?
  - pjmlp 5y ago
    Maybe it is my enterprise hat, but they always seem to miss something.
  - aunty_helen 5y ago
    I don't think you could attribute Trello to Atlassian. Maybe all the junk they've added to it. The other two I've honestly never heard of but that says nothing about them as products except they don't get talked about. Funny edit, Atlassian actually made a plugin for Jira called greenhopper that was very much like trello. But they discontinued it only to realise a few years later that they liked trello and paid a massive amount of money for it. If that doesn't exemplify incompetence of a companies strategy / directorship, then I don't know what does.
- pleb_nz 5y ago
  What caught up with JIRA I have yet to find something as good, even with all of JIRAs pain points
  - pas 5y ago
    Asana. It has its own idiosyncrasies, but it's faster, cleaner, supports things out of the box that I just gave up on in JIRA...
    - byefruit 5y ago
      Having used Asana in several organisations, I think it being a suitable replacement for JIRA says more about how poor a choice JIRA was for situation than anything about Asana.
    - Nextgrid 5y ago
      I've tried it before, it's just as bloated as Jira and performance suffers as a reason. If I'm going to suffer slowness anyway I might as well stick with Jira.
    - pleb_nz 5y ago
      Ive tried Asana breifly a few years agoand it didn't seem to have enough to make it worth the effort to swap. Maybe I should check it out again
      - pas 5y ago
        I don't recommend anyone to swap from JIRA. Stability is probably more important for a PM tool, but for new projects trying out something new makes sense.
  - Tehchops 5y ago
    We've been using https://linear.app/ https://linear.app/
- BlargMcLarg 5y ago
  >Jira itself has faded and been caught up with too. Tell this to all the big corps still running a slow, bureaucratic JIRA system demanding one to update multiple tickets multiple times.
ocdtrekkie 5y ago
It's an important highlight that Atlassian is deprecating all of their on-premise offerings and forcing everyone to the cloud because it's "the future". And then promptly demonstrated how poor of stewards of the cloud they are.
- noicebrewery 5y ago
  - ocdtrekkie 5y ago
    I don't have strong opinions on Atlassian, though I have strong opinions on forced cloud migrations.
  - obphuscate 5y ago
    it isn't nonsense you already can't buy new server products. Been like that since Feb 2021 and support ends entirely Feb 2024. https://www.atlassian.com/migration/assess/journey-to-cloud https://www.atlassian.com/migration/assess/journey-to-cloud. Try not to defend companies that don't deserve it.
    - noicebrewery 5y ago
      - obphuscate 5y ago
        Have you seen the prices on Data Center edition? People on server literally won't be able to afford it.
      - terom 5y ago
        Please take the 500 user minimum license and $42k/year cost into account. https://www.atlassian.com/licensing/data-center#data-center-approved-apps https://www.atlassian.com/licensing/data-center#data-center-...
- 5y ago
synicalx 5y ago
Good on them for doing a relatively detailed, honest and public review like this. But goddamn that recovery time is absolutely unacceptable for a modern mission-critical product like this, especially since they've essentially forced everyone onto their cloud platform. Not having the ability to automatically restore backups in 2022, when you're a MAJOR SaaS provider that stores massive amounts of essential data is just mind boggling to me. Hopefully this event has enough of an impact for them internally (and financially) that they seriously reevaluate things and maybe invest in some SRE capability, and better tooling.
- wryun 5y ago
  I think what they're implying is that they can restore _everyone's_ data quickly (i.e. 'usual' infra failure), and one Jira instance's data fairly quickly (but some manual steps), but the problem is they nuked a large subset of customers while everything else kept running. This means that they can't do a complete automatic restore, and instead have to deal with it customer at a time. I'm sure that they're going to make their customer at a time thing properly automatic now... Still a big failure, but a little more subtle than you make out. (disclaimer - worked for Atlassian years ago)
- terom 5y ago
  The list of affected data that was wiped out by the compliance delete and now requires recovery also includes the metadata in their orchestration system, and what sounds like other multi-tenant data stores. It's not just restoring a single, entire PostgreSQL database from backups. Lesson learned is not not have any semi-automated script dependent on human input/communication/decisions without also having a recovery plan for the worst-case scenario of said script being run with the wrong inputs.
- taeric 5y ago
  I agree that the timeline is not really acceptable. I am not sure anything mission critical should be hinged off a ticketing system, though. At best, things should be reported to the ticketing system, but never triggered from it as some sort of source of truth.
- deepspace 5y ago
neilv 5y ago
> Additionally, the majority of restored customers have had no data loss, while some have reported data loss for up to 5 minutes prior to the incident. This phrasing is ambiguous on a major question. If the reality is that no customers had more than 5 minutes' data loss from prior to the incident, then it'd be good to say that.
- jay_kyburz 5y ago
  I'll just assume some customers lost a lot more :) Not to mention all the data that was lost, because you can't create it, because the servers are down.
taspeotis 5y ago
> Faulty script. Second, the script we used provided both the "mark for deletion" capability used in normal day-to-day operations (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. The script was executed with the wrong execution mode and the wrong list of IDs. Bad script! Bad! So faulty, running itself with the wrong list of IDs and running itself in the wrong mode.
- mdaniel 5y ago
  I'd guess they mean in in the "blameless post mortum" way, in that it wasn't some Random User's fault, it was a total process breakdown that led to said hands-on-keyboard executing with the wrong information and safeguards in place
- karmakaze 5y ago
  This is what happens when DRY gets taken too far, deduplicating similarly named or conceptualized operations with vastly different use-cases.
- kadoban 5y ago
  It helps to write post mortems like this. If you start writing them in a way that suggests blame, you miss out on process issues and everyone clams up the next time it's post mortem time.
  - taspeotis 5y ago
    “A script was executed incorrectly” does not blame anyone. Saying a script was faulty obfuscates what happened.
    - kadoban 5y ago
      It is faulty though? There should be more that's required to accidentally delete data than a manually specified switch.
McTossOut 5y ago
yegle 5y ago
Wait I'm supposed to see a post? All I'm seeing is: Something went wrong. We're moving mountains to get it sorted. View our status page and subscribe for service updates.
murxmaster 5y ago
https://www.atlassian.com/engineering/april-2022-outage-update https://www.atlassian.com/engineering/april-2022-outage-upda... "Something went wrong. We're moving mountains to get it sorted." Works as axpected.
- triggercut 5y ago
  I just got the same! More than a bit worrying. Edit: After reviewing the snippets below looks like the script has gained sentience.
technick 5y ago
Self hosted solution ftw...
- philpem 5y ago
  Sadly not an option - Atlassian have stopped selling on-prem licences.
  - originalvichy 5y ago
    Not true. They only stopped selling the server edition (single node). Data center (single node or multi node) edition still exists.
    - cube00 5y ago
      Assuming you can afford the massive price hike from the current on prem pricing.
_pdp_ 5y ago
What will be the best way to backup and restore cloud instances of Jira and Confluence?
kingcharles 5y ago
> Instead of providing the IDs of the intended app being marked for deactivation, the team provided the IDs of the entire cloud site where the apps were to be deactivated. This reminds me of the time I was tasked with changing the gender of a user to female on the production database (since the feature was not available through an interface at the time) and so I opened SQL Explorer on our live database and ran a quick update. Except I forgot the UserID param and accidentally changed over a million registered users to female. A colleague came up with the idea to use the person's title as a hint to repopulate the database, which worked for Mr, Mrs and Ms. I apologise now to all the doctors that found they had undergone nonconsensual gender reassignment.
- wdb 5y ago
  If one record needs to be changed, I typically, just manually do it via the UI :)
- petepete 5y ago
  All databases should offer an --i-am-a-dummy mode, and probably enable it by default in clients. It prevents you from running update or delete statements that don't have a where clause. https://dev.mysql.com/doc/refman/8.0/en/mysql-tips.html#safe-updates https://dev.mysql.com/doc/refman/8.0/en/mysql-tips.html#safe...
- chrisandchris 5y ago
  That's why some databases have a "where condition required on update" so you won't hit that wall (assuming there were no other where conditions involved).
__app_dev__ 5y ago
Question for Atlassian Customers on this site. Does this event make you re-consider using their software? Also, I'm not clear if the site was down for all customers for a few days and now only 400 (0.18%) have an issue or if it was only down for only 400 customers. After looking at many tweets, comments here and reddit it looks like it was down for everyone but I'm not sure.
- ShinTakuya 5y ago
  It was down only for the 400. But some of them were bigger than others. In general though, it's natural for the affected customers to he more vocal than the unaffected customers, and given a large portion of the target market use Reddit/Hacker News, it's to be expected that you'll see a lot of comments.
  - __app_dev__ 5y ago
    Thanks!
contravariant 5y ago
Interesting, looks like my original estimate that they were trying to restore each customer individually a backup wasn't even that far off. Still odd that they didn't have any tools in place to help them with it. Only having a way to restore an entire backup is only useful for disaster recovery, it's nigh useless to correct mistakes.
everfrustrated 5y ago
The report is deliberately obscure. My interpretation is... Atlassian Cloud is hosted in shards where a shard is allocated a single AWS RDS database with (say) 100 customers. Their script was run improperly and dropped tables from 400 customers across 400 different shards (across different databases) Their backup strategy was just to restore RDS backups (recover whole database with many customers data together) But they can't simply restore from RDS backups due to the co-mingling of customers. As otherwise unaffected customers would now lose data. I suspect they're manually restoring RDS backups to new DB instances and having to scrape out the data for a single customer - a process they'd not built any tooling for as it was a fault case they thought they didn't need. Possibly as running the Jira/confluence backups are very heavy and create a lot of load/disk that they can't easily accomodation in a multi tenant hosting architecture.
- 9935c101ab17a66 5y ago
  Interesting hypothesis, and if true, I have the utmost empathy for the engineering teams that have to figure this out a way to kludge all the right data together. The thought of extracting the lost data from the sharded backups, and then merging it into a possibly a shard with good, newer customer data makes my skin crawl.
- boboche 5y ago
  If their backup strategy is as efficient as the UX of their tools, they should find what they are looking for in 2 weeks… oh wait… At least right now they are focussed on this issue and that should stop from outputting yet more unwanted bloat in my daily ops for a while. Guess its time to pay Asana another visit to see where they are at. Didn’t feel it was worth the move 2 years ago, maybe things have changed.
defaultcompany 5y ago
This ID thing makes no sense. Did the team doing the deletion take these cloud site IDs and plug then into a script expecting app IDs? And it still found and deleted sites? Or is the script like a delete anything script? Give it an ID and it will go find it in any table and delete it? That seems to be what they are implying.

132 points by taspeotis 5y ago

“End-to-end, it takes between 4 and 5 elapsed days to hand a site back to a customer.” Cool marketing strategy. Maybe you’ll get your data in a business week.

aunty_helen 5y ago

After listening to in-person talks from Atlassian in the past, I got the impression of them having a smugness from being a big fish in a small pond (Australian tech.) Their original and early Jira product was great and put them out in front, but I couldn’t name any other product of theirs except Bitbucket which is very much a second place competitor in their market. Jira itself has faded and been caught up with too. Reading through the post and seeing the level of support some people on Twitter have been tweeting about, I would take this as a great time to look at other offerings out there.

jeffybefffy519 5y ago
I think Confluence is a genuinely good product. Its nice to see that even Atlassian struggle with day to day sysadmin of Atlassian tools
- karmakaze 5y ago
  A lot of people look down on Jira. My experience was that Confluence was much worse. It turns writing a couple technical notes into a publishing journey with missteps and failures along the way.
  - VTimofeenko 5y ago
    For me Confluence dramatically improved in experience when I learned that it's possible to write files locally using $FAVORITE_EDITOR in confluencewiki format (or markdown if you don't care about macros) and post them through API.
    - clhodapp 5y ago
      Can you share anything more about your technique? As far as I know, Confluence hasn't supported a markup format natively for several years. I've found some tricks to allow me to author stuff in Markdown but it seems to come with a lot of limitations which ultimately mean that I have to start using the web editor as soon as I want to start collaborating with others.
      - DocTomoe 5y ago
        In Confluence Editor, under the Macro "Plus" button, you should find an entry called "Markup" pretty far up the list, which accepts "Confluence Wiki" Markup (which is just Markdown + Macros). This works for both on-premise as well as cloud instances.
      - VTimofeenko 5y ago
        I wrote a tool that automates the posting part[1]. Unfortunately I could not solve the collaboration problem either. Setting the body of the page overwrites it. I did see some examples of editing HTML through similar approach but never tried it myself. [1]: https://github.com/VTimofeenko/confluence_poster https://github.com/VTimofeenko/confluence_poster
- RexM 5y ago
  I read somewhere recently that confluence is write-only because search is so bad nobody will ever find what your write. I haven’t used it in years, but that seems accurate. I could never find anything unless I knew exactly how it was worded in what I’m looking for.
  - djbusby 5y ago
    Putting markdown text into SQLite with FTS5 works amazingly well for search. How come they couldn't just bolt that in?
  - wink 5y ago
    Yes, it's actually so bad that you wonder what part of the picture we're missing.
  - xen0 5y ago
    My experience with Confluence's search was that it's a mistake the press Enter. That is, the pop-up results that appear as I typed where significantly better than the actual "search results" page.
    - junon 5y ago
      I also vaguely remember this being the case. Confluence always felt so bloated to me.
- philliphaydon 5y ago
  When they first acquired confluence it was great. Now I can’t stand it. Is so slow now. So clunky.
jay_kyburz 5y ago
I had Jira inflicted on me before Atlassian was big and Jira was terrible back then too. I always liked FogBugs.
- macspoofing 5y ago
  >I always liked FogBugs You lost me there. In no way was FogBugz a better product. It was at best mediocre by 2010.
  - LilBytes 5y ago
    Current user of FogBugz, and prior user of JIRA. Jira is much better. We also use SharePoint and Confluence is so much better. Honestly I get the criticism of both JIRA and Confluence but till you've used the other big names in that space you don't realise how bad it can get.
    - jay_kyburz 5y ago
      Fogbugz was on the downhill as early as 2010. I always assumed that when Fogcreek started working on Trello they pulled the A team off Fogbugz.
    - macspoofing 5y ago
      >Current user of FogBugz I'm so sorry. Fogbugz has been in maintenance mode for years now, stuck in the middle of a half-finished redesign.
synicalx 5y ago
Having dealt with the pre and post sales support quite extensively over the years, I always got the impression they were a big fish in way over it's head, and with a tail that didn't work so good.
lbotos 5y ago
Bitbucket was also an acquisition (Most of their offerings were acquisitions). https://techcrunch.com/2010/09/29/atlassian-buys-mercurial-project-hosting-site-bitbucket/ https://techcrunch.com/2010/09/29/atlassian-buys-mercurial-p...
zeckalpha 5y ago
Trello? OpsGenie? Statuspage?
- pjmlp 5y ago
  Maybe it is my enterprise hat, but they always seem to miss something.
- aunty_helen 5y ago
  I don't think you could attribute Trello to Atlassian. Maybe all the junk they've added to it. The other two I've honestly never heard of but that says nothing about them as products except they don't get talked about. Funny edit, Atlassian actually made a plugin for Jira called greenhopper that was very much like trello. But they discontinued it only to realise a few years later that they liked trello and paid a massive amount of money for it. If that doesn't exemplify incompetence of a companies strategy / directorship, then I don't know what does.
pleb_nz 5y ago
What caught up with JIRA I have yet to find something as good, even with all of JIRAs pain points
- pas 5y ago
  Asana. It has its own idiosyncrasies, but it's faster, cleaner, supports things out of the box that I just gave up on in JIRA...
  - byefruit 5y ago
    Having used Asana in several organisations, I think it being a suitable replacement for JIRA says more about how poor a choice JIRA was for situation than anything about Asana.
  - Nextgrid 5y ago
    I've tried it before, it's just as bloated as Jira and performance suffers as a reason. If I'm going to suffer slowness anyway I might as well stick with Jira.
  - pleb_nz 5y ago
    Ive tried Asana breifly a few years agoand it didn't seem to have enough to make it worth the effort to swap. Maybe I should check it out again
    - pas 5y ago
      I don't recommend anyone to swap from JIRA. Stability is probably more important for a PM tool, but for new projects trying out something new makes sense.
- Tehchops 5y ago
  We've been using https://linear.app/ https://linear.app/
BlargMcLarg 5y ago
>Jira itself has faded and been caught up with too. Tell this to all the big corps still running a slow, bureaucratic JIRA system demanding one to update multiple tickets multiple times.

ocdtrekkie 5y ago

It's an important highlight that Atlassian is deprecating all of their on-premise offerings and forcing everyone to the cloud because it's "the future". And then promptly demonstrated how poor of stewards of the cloud they are.

noicebrewery 5y ago
- ocdtrekkie 5y ago
  I don't have strong opinions on Atlassian, though I have strong opinions on forced cloud migrations.
- obphuscate 5y ago
  it isn't nonsense you already can't buy new server products. Been like that since Feb 2021 and support ends entirely Feb 2024. https://www.atlassian.com/migration/assess/journey-to-cloud https://www.atlassian.com/migration/assess/journey-to-cloud. Try not to defend companies that don't deserve it.
  - noicebrewery 5y ago
    - obphuscate 5y ago
      Have you seen the prices on Data Center edition? People on server literally won't be able to afford it.
    - terom 5y ago
      Please take the 500 user minimum license and $42k/year cost into account. https://www.atlassian.com/licensing/data-center#data-center-approved-apps https://www.atlassian.com/licensing/data-center#data-center-...
5y ago

synicalx 5y ago

Good on them for doing a relatively detailed, honest and public review like this. But goddamn that recovery time is absolutely unacceptable for a modern mission-critical product like this, especially since they've essentially forced everyone onto their cloud platform. Not having the ability to automatically restore backups in 2022, when you're a MAJOR SaaS provider that stores massive amounts of essential data is just mind boggling to me. Hopefully this event has enough of an impact for them internally (and financially) that they seriously reevaluate things and maybe invest in some SRE capability, and better tooling.

wryun 5y ago
I think what they're implying is that they can restore _everyone's_ data quickly (i.e. 'usual' infra failure), and one Jira instance's data fairly quickly (but some manual steps), but the problem is they nuked a large subset of customers while everything else kept running. This means that they can't do a complete automatic restore, and instead have to deal with it customer at a time. I'm sure that they're going to make their customer at a time thing properly automatic now... Still a big failure, but a little more subtle than you make out. (disclaimer - worked for Atlassian years ago)
terom 5y ago
The list of affected data that was wiped out by the compliance delete and now requires recovery also includes the metadata in their orchestration system, and what sounds like other multi-tenant data stores. It's not just restoring a single, entire PostgreSQL database from backups. Lesson learned is not not have any semi-automated script dependent on human input/communication/decisions without also having a recovery plan for the worst-case scenario of said script being run with the wrong inputs.
taeric 5y ago
I agree that the timeline is not really acceptable. I am not sure anything mission critical should be hinged off a ticketing system, though. At best, things should be reported to the ticketing system, but never triggered from it as some sort of source of truth.
deepspace 5y ago

neilv 5y ago

> Additionally, the majority of restored customers have had no data loss, while some have reported data loss for up to 5 minutes prior to the incident. This phrasing is ambiguous on a major question. If the reality is that no customers had more than 5 minutes' data loss from prior to the incident, then it'd be good to say that.

jay_kyburz 5y ago
I'll just assume some customers lost a lot more :) Not to mention all the data that was lost, because you can't create it, because the servers are down.

taspeotis 5y ago

> Faulty script. Second, the script we used provided both the "mark for deletion" capability used in normal day-to-day operations (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. The script was executed with the wrong execution mode and the wrong list of IDs. Bad script! Bad! So faulty, running itself with the wrong list of IDs and running itself in the wrong mode.

mdaniel 5y ago
I'd guess they mean in in the "blameless post mortum" way, in that it wasn't some Random User's fault, it was a total process breakdown that led to said hands-on-keyboard executing with the wrong information and safeguards in place
karmakaze 5y ago
This is what happens when DRY gets taken too far, deduplicating similarly named or conceptualized operations with vastly different use-cases.
kadoban 5y ago
It helps to write post mortems like this. If you start writing them in a way that suggests blame, you miss out on process issues and everyone clams up the next time it's post mortem time.
- taspeotis 5y ago
  “A script was executed incorrectly” does not blame anyone. Saying a script was faulty obfuscates what happened.
  - kadoban 5y ago
    It is faulty though? There should be more that's required to accidentally delete data than a manually specified switch.

McTossOut 5y ago

yegle 5y ago

Wait I'm supposed to see a post? All I'm seeing is: Something went wrong. We're moving mountains to get it sorted. View our status page and subscribe for service updates.

murxmaster 5y ago

https://www.atlassian.com/engineering/april-2022-outage-update https://www.atlassian.com/engineering/april-2022-outage-upda... "Something went wrong. We're moving mountains to get it sorted." Works as axpected.

triggercut 5y ago
I just got the same! More than a bit worrying. Edit: After reviewing the snippets below looks like the script has gained sentience.

technick 5y ago

Self hosted solution ftw...

philpem 5y ago
Sadly not an option - Atlassian have stopped selling on-prem licences.
- originalvichy 5y ago
  Not true. They only stopped selling the server edition (single node). Data center (single node or multi node) edition still exists.
  - cube00 5y ago
    Assuming you can afford the massive price hike from the current on prem pricing.

_pdp_ 5y ago

What will be the best way to backup and restore cloud instances of Jira and Confluence?

kingcharles 5y ago

> Instead of providing the IDs of the intended app being marked for deactivation, the team provided the IDs of the entire cloud site where the apps were to be deactivated. This reminds me of the time I was tasked with changing the gender of a user to female on the production database (since the feature was not available through an interface at the time) and so I opened SQL Explorer on our live database and ran a quick update. Except I forgot the UserID param and accidentally changed over a million registered users to female. A colleague came up with the idea to use the person's title as a hint to repopulate the database, which worked for Mr, Mrs and Ms. I apologise now to all the doctors that found they had undergone nonconsensual gender reassignment.

wdb 5y ago
If one record needs to be changed, I typically, just manually do it via the UI :)
petepete 5y ago
All databases should offer an --i-am-a-dummy mode, and probably enable it by default in clients. It prevents you from running update or delete statements that don't have a where clause. https://dev.mysql.com/doc/refman/8.0/en/mysql-tips.html#safe-updates https://dev.mysql.com/doc/refman/8.0/en/mysql-tips.html#safe...
chrisandchris 5y ago
That's why some databases have a "where condition required on update" so you won't hit that wall (assuming there were no other where conditions involved).

__app_dev__ 5y ago

Question for Atlassian Customers on this site. Does this event make you re-consider using their software? Also, I'm not clear if the site was down for all customers for a few days and now only 400 (0.18%) have an issue or if it was only down for only 400 customers. After looking at many tweets, comments here and reddit it looks like it was down for everyone but I'm not sure.

ShinTakuya 5y ago
It was down only for the 400. But some of them were bigger than others. In general though, it's natural for the affected customers to he more vocal than the unaffected customers, and given a large portion of the target market use Reddit/Hacker News, it's to be expected that you'll see a lot of comments.
- __app_dev__ 5y ago
  Thanks!

contravariant 5y ago

Interesting, looks like my original estimate that they were trying to restore each customer individually a backup wasn't even that far off. Still odd that they didn't have any tools in place to help them with it. Only having a way to restore an entire backup is only useful for disaster recovery, it's nigh useless to correct mistakes.

everfrustrated 5y ago

The report is deliberately obscure. My interpretation is... Atlassian Cloud is hosted in shards where a shard is allocated a single AWS RDS database with (say) 100 customers. Their script was run improperly and dropped tables from 400 customers across 400 different shards (across different databases) Their backup strategy was just to restore RDS backups (recover whole database with many customers data together) But they can't simply restore from RDS backups due to the co-mingling of customers. As otherwise unaffected customers would now lose data. I suspect they're manually restoring RDS backups to new DB instances and having to scrape out the data for a single customer - a process they'd not built any tooling for as it was a fault case they thought they didn't need. Possibly as running the Jira/confluence backups are very heavy and create a lot of load/disk that they can't easily accomodation in a multi tenant hosting architecture.

9935c101ab17a66 5y ago
Interesting hypothesis, and if true, I have the utmost empathy for the engineering teams that have to figure this out a way to kludge all the right data together. The thought of extracting the lost data from the sharded backups, and then merging it into a possibly a shard with good, newer customer data makes my skin crawl.
boboche 5y ago
If their backup strategy is as efficient as the UX of their tools, they should find what they are looking for in 2 weeks… oh wait… At least right now they are focussed on this issue and that should stop from outputting yet more unwanted bloat in my daily ops for a while. Guess its time to pay Asana another visit to see where they are at. Didn’t feel it was worth the move 2 years ago, maybe things have changed.

defaultcompany 5y ago

This ID thing makes no sense. Did the team doing the deletion take these cloud site IDs and plug then into a script expecting app IDs? And it still found and deleted sites? Or is the script like a delete anything script? Give it an ID and it will go find it in any table and delete it? That seems to be what they are implying.