66 ms·

Inside the longest Atlassian outage

• avidphantasm 5y ago
  CTO should be fired.
• 8n4vidtmkvmk 5y ago
  hahaha, i left atlassian when they deleted everyone's mercurial repos with no way to export either. fuck them. they've proven time and time again that they dont care about their customers
• N19PEDL2 5y ago
  What are good alternatives to Jira and Confluence? Asking for a friend.
• nitinagg 5y ago
  Selectively restoring data only for certain rows is super hard. But the communications by Atlassian has been the worst I have ever seen in the industry.
  • profmonocle 5y ago
    I actually got an email from our Atlassian contact just the other day encouraging us to switch to their cloud service. Crazy that no one thought to pause those. (I assume it must have been scheduled.)
    • HeyLaughingBoy 5y ago
      This article on HN is the only time I've even heard that Atlassian was having a problem. I suspect that 99% of the tech "community" has absolutely no idea this is happening. We use Jira, but it's self-hosted for my team. Maybe other teams that have transitioned to the cloud version are aware that there's a problem, but I haven't heard about it.
      • mcintyre1994 5y ago
        It’s only 400 teams affected, but from this article it sounds like they’re all really big ones.
      • LadyCailin 5y ago
        Apparently the self hosted version goes out of support in 2024, so there will only be cloud hosting. Dumb dumb dumb.
  • duxup 5y ago
    As this continues I suspect that this might be one of the few times where a lack of transparency / good communication really ... might not be better or worse because the situation is so bad that transparency would be horrible just the same. Granted that's how all lies start / what sometimes people assume and they're wrong but ... maybe this is that time? Maybe it is in fact so bad that honesty would be a push or worse?
    • adamc 5y ago
      If so, that itself would be a huge red flag for dealing with Atlassian.
      • duxup 5y ago
        I think it is…either way.
    • williamscales 5y ago
      > Maybe it is in fact so bad that honesty would be a push or worse? In my opinion, such a scenario does not exist. Transparency always in all things.
  • tmpz22 5y ago
    It’s super hard no doubt but I wonder how much of the data was hot vs cold.
  • raincom 5y ago
    So, it must be a bad idea to shove the data of multiple customers in a single table controlled by some column name ('tenant').
  • ollien 5y ago
    As someone who has never had to perform this kind of recovery: why is it so hard?
    • jacquesm 5y ago
      Because it is very difficult to maintain relational integrity during a restore like that.
      • ollien 5y ago
        Gotcha. I guess you could be heavy-handed and disable foreign key checks, but who knows what other bugs that would bring into the mix.
        • teling2 5y ago
          The other difficulty is if you don't restore the entire state in a single transaction. Imagine you have partial data restored in Table A but haven't updated Table B correspondingly. Now some other program that consumes Table A and Table B and doesn't have error handling will crash (or worse, mutate state in other weird ways).
          • jacquesm 5y ago
            That is relational integrity.
  • jacquesm 5y ago
    It is, but between 'hard' and 'impossible' there is the nagging question of whether you actually really still have that data.
  • seanwilson 5y ago
    > Selectively restoring data only for certain rows is super hard. What's the right way to structure your data here that would make restoring more straightforward here? Is this backup/restore scenario niche or they should have designed for it?
    • inopinatus 5y ago
      in theory, shard your customer databases 1:1, job done. alas, in practice, many SaaS compromise this two ways: a) overwhelmed by creeping featuritis, each customer's data has relationships to global tables, and b) they backup their entire database cluster in one snapshot and there maybe other gotchas for restoration, like relying on denormalized views and caches that have to be rebuilt. they may also have erroneously assumed that data protection's main value driver is whole-of-system disaster recovery, which can lead to pathologies such as "we don't have a single-customer restoration tool". this is not a niche scenario
      • seanwilson 5y ago
        > shard your customer databases 1:1 What are the downsides to this?
        • inopinatus 5y ago
          * makes it much harder to distribute your tables by any other factor, for whatever reason (usually performance, sometimes archival) * disaggregates data that the SaaS might be interested in querying/updating as an aggregate * not all ORM frameworks handle this case well, if at all * dumps are more than a single trivial command basically all your data operations gain an additional dimension of complexity, and you may not perceive the benefits until much later
          • seanwilson 5y ago
            Would it be fair to estimate that the majority of SaaS companies aren't sharding like this then? Seems like a lot of downsides that impact everything often except for backups, which you'd restore rarely.
  • chousuke 5y ago
    If the database schema for Jira on the cloud is anything like the Datacenter version, I'm not surprised they're having a hard time restoring data. I once tried to figure out how to find duplicate / redundant project schemas by querying the database (the required APIs are cloud-only) and could not even find which tables stored half the data, never mind how they referred to each other.
  • miketria 5y ago
    Hi, this is Mike from Atlassian Engineering. You are right the communications from us have not lived up to our standard. We will focus on this specifically once we restore service and get the post incident review out there. More details here: https://www.atlassian.com/engineering/april-2022-outage-update https://www.atlassian.com/engineering/april-2022-outage-upda...
    • lallysingh 5y ago
      Spamming HN isn't helping your cause man.
      • Mysterise 5y ago
        There is irony in complaining about over-communication when it's in response to criticisms of under-communication.
        • 2muchcoffeeman 5y ago
          Well why are they writing a blog and posting the link on HN? We’re not directly your customers. Did you apologise individually to the customers you ignored? You don’t have to apologise to anyone here.
        • lallysingh 5y ago
          Key word "spamming." It wasn't communication but another dry and information-free blob of text. Communication requires something to say.
          • dhzhzjsbevs 5y ago
            It's worse than that, they're saying communication was not up to their standards without actually communicating anything we didn't already know. At least explain why there was such a total communication blackout company wide. Even support staff weren't allowed to discus it. Why?
      • deleted 5y ago
        [deleted]
  • anarazel 5y ago
    ISTM the fairly obvious approach would be to bring up a complete copy of the affected database(s) and move the affected tenants to that "copy", while eventually deleting non-affected tenants. Can't imagine they don't have the ability to move tenants to different shards, they got to need that to deal with quickly growing customers etc.
• h2odragon 5y ago
  > However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architecture choices that they're trying to hide. Not that I ever thought using their products was a good idea; to be clear about my position... But at this point anyone continuing to rely on them for anything is asking for the suffering they'll get. Signing up for their crap for a vital business function is like offering your tonker to a snapping turtle.
  • outworlder 5y ago
    > OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production This seems to be exactly what they are doing, as described in the article. They don't have automated tools to do this.
  • JoBrad 5y ago
    It sounds like that’s what they are doing, but it’s manual.
  • omoikane 5y ago
    Not being able to selectively restore data for a subset of users might also indicate that users are not fully isolated from each other, which is worrying for technical and nontechnical reasons.
    • indymike 5y ago
      There is nothing non-technical that matters. If we start acting like it does, we incredibly poor decisions that in fact have nothing to do with physical reality, and quickly arrive at unworkable technology.
      • omoikane 5y ago
        Non-technical reasons include "legal" and "compliance", which often matters a fair bit. I am not disagreeing that non-technical requirements occasionally lead to poor decisions, for some value of poor.
• mkl95 5y ago
  The fact it's been so long and they still haven't revealed and explained the root cause of the outage is going to make it hard to regain trust on their buggy, slow tools. The bright side of the incident is that competitors that somewhat care about users have a unique opportunity to stand out.
  • pgwhalen 5y ago
    > The fact it's been so long and they still haven't revealed and explained the root cause of the outage They did last night: https://www.atlassian.com/engineering/april-2022-outage-update https://www.atlassian.com/engineering/april-2022-outage-upda...
    • hu3 5y ago
      > Faulty script. Second, the script we used provided both the "mark for deletion" capability used in normal day-to-day operations (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for approximately 400 customers were improperly deleted. Ouch. I hope no one person got the blame. This is a systemic failure. Regardless, my regards to the engineers involved.
      • gtm1260 5y ago
        Right? The way this reads it seems like one person set a flag incorrectly, something I'm sure we've all done numerous times. And there were no checks down the line to catch it.
        • miketria 5y ago
          Hi, this is Mike from Atlassian Engineering. You are right that the checks need to improve to reduce human error, but that's only half of it. I don't see this as human error though. It's a system error. We will be doing some work to make these kind of hard deletes impossible in our system.
• luckydata 5y ago
  so this is the end of Atlassian as a company right?
  • raincom 5y ago
    I don't think so, as long as investors hold the stock, as long as customers keep paying Atlassian.
  • case0x 5y ago
    Why would it ? On our end everything works fine. If you’re not one of the 400 companies, there's no difference
    • asah 5y ago
      "first they came for ..."
      • gmfawcett 5y ago
        Poor taste, buddy. Comparing the Atlassian mess-up to the Holocaust diminishes the Holocaust.
        • asah 5y ago
          um... the sentiment is universal it's not specific to that particularly awful history. Sorry if it triggered you, HN doesn't offer a delete button. FYI my ancestors fled oppression on both sides and I'm well aware that it's a miracle I'm alive. Again, one bad thing leading to another is a common human behavior, and the Holocaust is just an extreme example that I ABSOLUTELY did not intend whatsoever. You make this connection, not me.
          • gmfawcett 5y ago
            If I'm then one making this connection, then it should be trivial for you to finish your sentence. "First they came for..." Who are the Jews in your analogy ? Who are the communists? the trade unionists? And who is the totalitarian regime? Suggesting that Niemöller's poem is about "one bad thing leads to another" is like suggesting that Anne Frank's diary is about "sometimes girls have really bad days." I understand you didn't mean any offense to anyone. But that's not a license to be offensive, and then duck for cover.
• R0ger 5y ago
  I guess this is wake call for the people rushing to SaaS solutions.
  • brianwawok 5y ago
    Is it? We use JIRA. Not impacted. If this had hit us.. we would just switch to excel or something for a week/month? But maybe we are a very light user of JIRA. Nothing in there can't be replaced. It's "nice" to be able to go look up a 3 year old bug and which client reported it, but not really crucial for day to day ops.
    • mrits 5y ago
      I wonder why you use Jira if a spreadsheet is sufficient for your use case.
      • brianwawok 5y ago
        It allows some simple links like story X depends on story Y, and displays it in a visually pleasent way.
      • HeyLaughingBoy 5y ago
        He didn't say it was sufficient; he said they could do it for a short while. I consider myself in the same situation: we depend on Jira, but for a week or so it's not a big deal to use a bunch of Post-It notes.
      • function_seven 5y ago
        Same reason I use oil lamps when the power is out, even though electric bulbs are my normal lighting. A spreadsheet may be sufficient, but it's not as good as a system designed for development workflows. (This comment sounds like I have a speck of love for JIRA. I don't! :)
        • mrits 5y ago
          I don't see this as a valid comparison. There is information loss. This has happened to my team which had about 50 people and it was very chaotic. It took us several days to just create the state our features were in. Today it would even be more troublesome as we have a lot of integration rules dependent upon the workflow. I'd probably just recommend everyone uses a few weeks for self improvement and only address critical production issues.
• hsnewman 5y ago
  Sounds like the continuity planners at Atlassian (the fall guys) will be looking for a new job.
• farseer 5y ago
  They have recently killed off on premise offerings, it's cloud only now. And this makes it harder to trust both the security and integrity of your data.
  • ocdtrekkie 5y ago
    The fact that a single bad script could delete 400 of their customers should be absolute proof they do not have the processes in place to be a steward of your data in the cloud. On-prem or bust.
    • dangus 5y ago
      On-premise just means that your overworked IT person is going to spend 5% of their time keeping your service maintained, at no point gaining any more than baseline familiarity with the product. On-premise isn’t a magic pill guaranteeing 100% uptime and 0 data loss. While on-premise may be a good choice in many cases, it’s not like running on-premise business tools has no risk associated with that choice. Remember that the goal of a company is to sell the most product possible (output) with the lowest cost possible (input). Any Joe off the street starting their own business can pay Atlassian $0/month for up to a 10 users. On-prem doesn’t compete with that.
      • dzikimarian 5y ago
        On Prem means you have control over spending. I calculated that if we've moved to the cloud, we would pay YEARLY as much as we spent on Atlassian licenses in last 5 years. That easily pays for the maintenance overhead on our devops team.
  • deleted 5y ago
    [deleted]
  • hrpnk 5y ago
    Afaik, the Data Center option still allows for on-premise deployment, incl. Kubernetes and cloud deployments [1, 2, 3]. [1] https://www.atlassian.com/enterprise/data-center https://www.atlassian.com/enterprise/data-center [2] https://confluence.atlassian.com/enterprise/jira-data-center-documentation-472219731.html https://confluence.atlassian.com/enterprise/jira-data-center... [3] https://confluence.atlassian.com/enterprise/deploying-enterprise-scale-jira-on-aws-a-step-by-step-guide-979435325.html https://confluence.atlassian.com/enterprise/deploying-enterp...
• rmbyrro 5y ago
  Are Confluence pages and Jira tickets build like a GPT-3 300 Terabyte model? I mean, I thought they were text. 5 days to restore text? They must be generated by a huge complex deep learning voodoo. Atlassian is working on the bleeding edge of technology. This outage is understandable...
  • er4hn 5y ago
    I suppose if they recover what they can and restore the rest using GPT-3 that may make the process easier.
  • katbyte 5y ago
    images and other files can be attached to issues or embedded in pages so a single instance can use a lot of storage.
• deleted 5y ago
  [deleted]
• jgrahamc 5y ago
  Communicate directly and transparently Yes. Always.
• politelemon 5y ago
  > it takes between 4 and 5 elapsed days to hand a site back to a customer. Atlassian's SLA page says, Premium Cloud Products 99.9% That's 43 minutes of downtime per month. That works out to, Atlassian can't have any more downtime for the next 14 years. Are SLAs even real? I'm being slightly facetious. From the page text it's just a threshold after which I think you're entitled to some money back for that month.
  • bborud 5y ago
    Think of SLAs as "this is how hard we'll scramble when shit hits the fan". Except...I don't even believe that.
    • chrsig 5y ago
      It's more "this is our contractual obligation, if we're down more than this, then we might not charge you"
      • dylan604 5y ago
        Lawyers are involved, so I'd assume some text about "excluding acts of god, sabotage,etc" to weasel their way out of things. They might even be able to get away with "acts of incompetence" how ever a lawyer might phrase that to allow their client to weasel.
        • TheCoelacanth 5y ago
          SLA credits are a thing that actually happen in the industry. I wouldn't automatically assume that they will be able to weasel out of it. They are typically limited to the amount that you actually paid, though, so basically they don't charge you for the time when you couldn't use the product. You usually won't get more than that.
        • mywittyname 5y ago
          That's a good way to get executive approval to replace a system. Google or Apple can get away with this kind of behavior, I doubt Atlassian can. This outage alone has spurred conversations in slack about how terrible JIRA is and why we should replace it. If this kind of shit was pulled, I can guarantee we'd be on shortcut, linear, or something else in short order.
• bogomipz 5y ago
  >"The outage is its 9th day, having started on Monday, 4th of April." >"It took until Day 9 for executives at the company to acknowledge the outage." Just to put this in perspective. These executives would have left on a Friday afternoon to start their weekends without bothering to publicly address an ongoing outage that was by then 5 days old. This is mind boggling. Like did some C-level exec say something like "Let's just park this whole outage communication discussion until Monday, have a good weekend everyone."?
• Alex3917 5y ago
  A few years ago we didn't renew our subscription on time because we got the email over Christmas break, and iirc they deleted all of our data in less than two weeks. They were eventually able to manually restore it from backups, but they restored it incorrectly so there was a bunch of stuff broken. This whole thing isn't even remotely surprising to me.
  • jacquesm 5y ago
    Did you continue as their customer after that?
    • Alex3917 5y ago
      Nope. I exported our data after they restored the backup and then we cancelled less than a month later. Like I obviously understand suspending our logins, but why would you ever delete someone's data when it's literally only 160 KB of text? The whole thing made zero sense.
      • Kwpolska 5y ago
        > why would you ever delete someone's data when it's literally only 160 KB of text? Compliance? The contract has expired, so there’s no legal basis for them to keep your data?
        • usefulcat 5y ago
          Seems like that could be addressed with some fine print in the initial agreements. "In the event that you stop paying us, we may keep your data for up to N days unless directed otherwise by you"--or similar.
          • bzxcvbn 5y ago
            Why would they bother?
        • deleted 5y ago
          [deleted]
• nemothekid 5y ago
  >Most of them said they won’t leave the Atlassian stack, as long as they don’t lose data. This is because moving is complex and they don’t see a move would mitigate a risk of a cloud provider going down. I still don't understand the strangehold JIRA has on some clients. I can't quickly think of another SaaS product that could be down for almost 2 weeks and not have most customers leave.
  • z58 5y ago
    I imagine most people used something like Google Sheets during the downtime
  • chupchap 5y ago
    A lot of companies have integrations to atlassian suite which might not be easy to shift from. Secondly, there are a lot of individual competitors to Jira, Confluence and Bitbucket but which competitor can offer all three under a single invoice? May be Microsoft, can't think of anyone else. Also for such an extended downtime the customers are entitled to a discount or a credit note which a lot of CXOs consider in their decision making.
    • judge2020 5y ago
      > May be Microsoft, Is there a Jira replacement/offering in the Microsoft 365 suite?
      • travellingprog 5y ago
        Never used it, but looks like Microsoft Project would fit that box.
        • HeyLaughingBoy 5y ago
          Oh god, no. We moved from Project to Jira and life became immeasurably better!
      • ralgozino 5y ago
• cdjk 5y ago
  This isn't the longest outage - last time they couldn't recover and recovered data from email archives.
• ChrisMarshallNY 5y ago
  Heh. We have a Confluence account. That no one uses. So we didn't notice.
  • teh_klev 5y ago
    You probably wouldn't if you weren't in the affected subset of customers who were. This wasn't a total outage, but rather it affected a group of users who had been running a legacy standalone app called "Insight – Asset Management".
• throwawayHN378 5y ago
  When in doubt I go on LinkedIn and find an engineer that works for the company and message them directly.
  • rjmunro 5y ago
    Does that work? I've sometimes thought about trying it, but never actually done so.
• 1970-01-01 5y ago
  Interesting note: Atlassian stock (NASDAQ: TEAM) is up 4% as of noon today.
  • radicaldreamer 5y ago
    It might be a good short opportunity... I imagine a lot of customers are kicking off their own internal process for migrating away from JIRA. By the time they actually do, it'll be at least a couple of quarters from now, which is when the customer hit will start materializing in quarterly results for the company. Maybe time to throw a few chips at some long term puts?
    • eli 5y ago
      Aren't most customers in 12+ month contracts? A migration seems like it would take many months to select a new vendor and migrate regardless. Be careful about the date on those puts. It's pretty hard to out-think the market on this kind of stuff. I'd just as soon bet the other way: few customers will actually churn and in 6 months this won't really matter.
      • __app_dev__ 5y ago
        They might even get some new customers after people who never used it look at their site and offerings. Disclaimer I have puts that expire 4/22 (purchased yesterday) so I hope they go down in the short term. Seems like a total loss now after being up 50% yesterday.
    • Iolaum 5y ago
      Given how out of sync tech people are with the general population I 'd be tempted to think it's a buy opportunity. Time will tell.
    • dahdum 5y ago
      I wouldn’t short. They just slapped 400+ customers and likely hundreds of thousands of users in the face and the C-suite didn’t think it was important to even acknowledge. That might look like incompetence, but I think it’s confidence. They know the switching costs for large orgs are so high they can treat these people like trash and few if any will leave. I wouldn’t be surprised if the total number of seats among affected customers has gone up in a few months. By failing to acknowledge the problem they’ve kept it out of the mainstream media and financial press. They have their customers by the balls and don’t respect them. That’s a short term bullish signal to me.
• a-dub 5y ago
  i hate deleting things. prefer flags that hide things instead (like a boolean deleted flag in an rdbms table). prevents data integrity issues in relational databases, makes debugging easier and prevents disasters. ideally also include a timestamp, both for bookkeeping and safe tools that only remove things that have been soft deleted for some time and are safe to delete without compromising integrity of anything that is not deleted (this is especially important in relational data models)
  • bombcar 5y ago
    The GDPR and various things have made companies more skittish in doing things this way, because they get scared. Perhaps an effective measure would be to create a key that encrypts a customer's data, and give them a copy of the key, and let them know that after a certain point your copy of the key will be deleted, and if they want a restore past that point they'll need to provide the key.
    • brimble 5y ago
      You may as well just delete it, then. I guarantee a high percentage of users won't save that key and be able to find it later. GH (edit: or similarly nerdy sites) might (might!) be able to get away with that, but as soon as part of your process is "give the user a cryptographic key" you've just guaranteed yourself a support nightmare, with normal users. It's why the only cryptographic person-to-person communication systems that've been broadly successful haven't involved keeping track of anything, and don't have a setup process more complex than "point camera at QR code".
      • bombcar 5y ago
        Yeah, you end up in the case where you "officially" cannot recover after X, but then you make sure that "accidentally" you might be able to recover by keeping copies around somewhere ... until someone realizes and you get sued.
    • a-dub 5y ago
      that's an interesting question, i've given a little thought to this multi tenant saas stuff... not sure if the right way forward is some sort of innovation in operating system and software design where people write and run apps that feel like single tenant apps attached to dedicated per tenant datastores where os and framework magic handle per tenant encryption and segmentation (tenant id as an os level concept) or... if it makes more sense to encrypt at the record level with keys that only the customers hold using (assuming it's up to the task) homomorphic encryption for things like searches and other backend functions. either way, for now, soft deleting and following up with an automatic daily hard delete of things soft deleted more than x days ago is a totally reasonable approach. ops scripts should require typing "yes i know what i'm doing" if someone attempts to hard delete things that have not yet been soft deleted.
• deleted 5y ago
  [deleted]
• scottlamb 5y ago
  Gmail had a vaguely similar outage years ago. [1] tl;dr: 1. Different root cause. There was a bug in a refactoring of gmail's storage layer (iirc a missing asterisk caused a pointer to an important bool to be set to null, rather than setting the bool to false), which slipped through code review, automated testing, and early test servers dedicated to the team, so it got rolled out to some fraction of real users. Online data was lost/corrupted for 0.02% of users (a huge amount of email). 2. There were tape backups, but the tooling wasn't ready for a restore at scale. It was all hands on deck to get those accounts back to an acceptable state, and it took four days to get back to basically normal (iirc no lost mail, although some got bounced). 3. During the outage, some users could log in and see something frightening: an empty/incomplete mailbox, and no banner or anything telling them "we're fixing it". 4. Google communicated more openly, sooner, [2] which I think helped with customer trust. Wow, Atlassian really didn't say anything publicly for nine days?!? Aside from the obvious "have backups and try hard to not need them", a big lesson is that you have to be prepared to do a mass restore, and you have to have good communication: not only traditional support and PR communication but also within the UI itself. [1] https://static.googleusercontent.com/media/www.google.com/en//appsstatus/dashboard/ir/nfed4uv2f8xby99.pdf https://static.googleusercontent.com/media/www.google.com/en... [2] https://gmail.googleblog.com/2011/02/gmail-back-soon-for-everyone.html https://gmail.googleblog.com/2011/02/gmail-back-soon-for-eve...
  • fishnchips 5y ago
    Funny enough, most of what we restored then was spam (ex gTape SRE, remember the outage).
    • fletchowns 5y ago
      Even though you are no longer there...I had a friend who recently had her gmail inbox mysteriously emptied, all emails seemingly permanently deleted. She paid for Google One to be able to talk to support, and they said that the data is gone. Do you know if there's a way to recover this data? She is quite heartbroken at all the attachments that she will never get to see again.
• jmondi 5y ago
  What blow's my mind is that Atlassian stock has barely taken a hit...
  • pigtailgirl 5y ago
    the stock is on a rally today - just goes to show - the market is full of surprise
  • capableweb 5y ago
    It was a long time ago individual stocks represented anything grounded in reality. People talk about "fundamentals" and so on, but that's not what the price is based on. I don't think anyone know why the prices move as they do anymore, as there are so many algorithms involved today, both manual and automatic ones.
  • devmunchies 5y ago
    it was at $317 on the day of the outage and now at $278.5. A ~12% drop. You're right, not much of a drop for such a large outage.
    • deleted 5y ago
      [deleted]
    • __app_dev__ 5y ago
      The outage did not impact the stock, most major tech stocks have taken a large hit in the past week and a half (until today). This even is not even showing on any financial news site. I'm still hoping it does and the stock goes down because I place an option order yesterday betting that it goes down by next Friday. Seems like it won't now but the risk was worth taking in my book.
  • NineStarPoint 5y ago
    Unless their revenue takes a long term hit over the outage, no reason for the stock market to care. There isn't news of people actually planning to stop using Atlassian products over this. The only direct consequence is going to be the one time payment of SLA credits. So I guess the part I find surprising is how little impact this looks like it will have on people using their products more so than I am that the stock market doesn't care much about this.
• Vaslo 5y ago
  I bet the Shitlassian guy is dancing and singing because of this.
• jacquesm 5y ago
  I suspect - pure speculation - that they can't restore the backups, because if they could then they could easily do this in a way that accounts affected could be restored selectively. In other words: test your backups, if you don't they won't be there for you when you need them.
• Aissen 5y ago
  The sad truth is that with 99.8% of customers unaffected, it was probably thought to be a minor issue. If those customers didn't have Gergely's ear we probably wouldn't have heard about it.
  • tpmx 5y ago
    Is there a source on this number?
    • Aissen 5y ago
      From the article: > Atlassian claims the customers impacted were “only” 0.18% of its customer base at 400 companies. From https://jira-software.status.atlassian.com/ https://jira-software.status.atlassian.com/ : > The team is continuing the restoration process for the ~400 impacted customers.
      • tpmx 5y ago
        > The team is continuing the restoration process for the ~400 impacted customers. We have restored functionality for 45% of impacted users. If this is truthful it implies implies more than 400 impacted customers.
        • zmmmmm 5y ago
          even in the middle of the crisis they will play with ambiguity around the words "user" and "customer" to obfuscate the real situation (presumably they don't consider users 'customers', which goes right to the heart of enterprise software in general)
  • miketria 5y ago
    Hi, this is Mike from Atlassian Engineering. Not a minor issue. Once we knew the extent and severity of the incident, we had hundreds of engineers engaged and working to restore service.
    • 5y ago
• kgeist 5y ago
  We use on-premises setups for almost everything (we generally avoid cloud solutions to have full control of our data), sometimes (approximately once a month) it goes down for a few minutes which already feels like a torture because all our processes depend on it, I can't imagine having no access to it for several weeks, all our work would stop to a halt... The office of the guy who administers on-premise servers is literally next door, all it takes is to make a visit to him and everything works again after 5 minutes. Reading horror stories like this (Slack being down, Atlassian being down, no one knows what is happening and when it will end etc.), I wonder why many companies choose cloud solutions for critical business processes. Is it pricing? Ease of use? I can understand why very small companies would choose it, but I don't understand why a medium/large business would choose anything but an on-premises setup.
  • lloydatkinson 5y ago
    Cloud solutions can work well. I've used GitHub, Azure Devops, and BitBucket (another wonderful atlassian product /s) and BitBucket frequently craps out, multiple times a week. We need to rerun builds in TeamCity because BitBucket stops talking to it.
    • tgmatt 5y ago
      Bitbucket is so slow and unreliable for us we regularly refer to it as Shitbucket. About as useful as a bucket of crap most of the time.
  • Msw242 5y ago
    You're assuming every team would have better uptime with in-house solutions I think many would have worse uptime even with more headcount
    • kgeist 5y ago
      I've heard this argument many times before, but is there research into this? I.e. where they would compare uptime of cloud vs. on-premises across a wide range of companies.
• bluedino 5y ago
  Regarding the backup restores: I once worked a company that had a data loss issue. There was nothing else we could do, we had exhausted every option we had over almost 40 hours. At the end of the second day, it was decided to restore from backup. We had done this before, as a test. It took about 12 hours to restore the data and another 12 hours to import the data and get back up and running. One small thing was different this time, and it had huge consequences. As a cost-saving measure, an engineer had changed the location of our backups to the cold-storage tier offered by our cloud provider. All backups, not just 'old' ones. This added 2 additional days to our recovery time, for a total of five days. Interestingly enough, even though we offered a full month's refund to all of our customers, not even half of them took us up on it.
  • bradknowles 5y ago
    Did I ever tell you guys about the time we accidentally nuked all the mailboxes for all the million-plus users on The Global Network Navigator (GNN) site? And how the restore process failed for us? This hasn't been written up at The Register yet, so I don't have a single URL I can share with you.
  • bombcar 5y ago
    In these cases the best thing to do is just give every customer the full month refund; don't make them ask for it.
    • sodality2 5y ago
      The best thing to do business-wise, or as a good faith move?
      • rjmunro 5y ago
        What's the difference?
        • treesknees 5y ago
          Good faith would be to lose all of that money to people who are already your customers. Business-wise would be to stay in their good graces and keep those customers by offering the refund, but you don't lose any money to those who either don't care or won't move to a competitor.
• napolux 5y ago
  Yeah, let's centralize the Internet (born decentralized). This is what the Internet has become.
  • Crabber 5y ago
    How do we solve this problem? In other industries based on physical products there is a big incentive to buy goods as locally as possible because of reduced shipping costs, shorter shipping time, no import taxes etc. But with software it costs nothing to spin up new instances, costs nothing to deliver half way across the world, and has no delivery time. How can you convince a manager to use a software solution provided by a local company when a company in a completely different country 600 miles away offers similar software with 5 extra features? It seems like the internet is now perfectly set up to create, for each software type, a single company that has a global monopoly.
    • barneygale 5y ago
      That's OK in principle, as long as those companies function like governments (i.e. they work to improve things rather than turn a profit, subject to constitutions, public voting, judicial review). As engineers we should embrace the efficiency of scale, but it's quite clear that it can't work under capitalism.
• ineedasername 5y ago
  Something to consider is that Jira can require a great deal of configuration to tailor it to your needs. If you already have a DevOps team of some capacity (not everyone does) then it may only be a small incremental increase to run thinks on prem. I did it myself: I'm ver much not a DevOps person, mostly unfamiliar with optimizing JVM parameters for apps like this, but it still only took me about 5 hours to get things running stable, and then another 2 hours or so a few weeks later to tweak things like heap size to help things go a bit faster (though it was still somewhat slow) To be complete open though I don't know how much DevOps overhead is involved in maintenance or feature updates. I hated the app and used it for less than a year so I didn't have much exposure. I guess my point though is simply that you may not need to use their SaaS option if you have a decent DevOps team already. After the initial setup time I doubt I spent more than half an hour a month managing the internals and updates. I did spend more than that on configuring the system for use, which you'll need to do regardless.
  • thesh4d0w 5y ago
    Atlassian has EOL'ed their non-cloud products https://www.atlassian.com/migration/assess/journey-to-cloud https://www.atlassian.com/migration/assess/journey-to-cloud
    • originalvichy 5y ago
      I have had to correct this too many times already. Server is the name of the deployment type of their on-prem. It means single node non-clustered. Data center is their deployment that supports clustering to multiple nodes (and used to support a few extra features). They are retiring the Server deployment type licenses and pushing everyone to data center or cloud. So no, they aren’t EOLing their on-prem.
      • bombcar 5y ago
        The cost of server was lower and fixed (buy it once), the cost of datacenter is MUCH higher (minimum 500 users, pay per year). Which is even more amusing when you realize Server has been Datacenter with a fake mustache for years now.
• hinkley 5y ago
  The longest Atlassian outage so far…
• snarkerson 5y ago
  > Most of them said they won’t leave the Atlassian stack, as long as they don’t lose data. This is because moving is complex and they don’t see a move would mitigate a risk of a cloud provider going down. However, all customers said they will invest in having a backup plan in case a SaaS they rely on goes down. The real key lesson here. Your business is important to you. Not so much to the service provider.
• flaviotsf 5y ago
  I recommend doing disaster recovery steps for your personal data as well, such as Gmail. At one point recently I was creating filters to delete bulk messages and - when the filter got created, it somehow missed the from:@xyz.com domain part and I ended up deleting => delete forever all emails. I noticed the issue right away but it was enough to wipe 2-3 months worth of emails (all of them, even Sent ones).
• oldshatterhand 5y ago
  Random guess, that this is a "we say we make backups, but we actually take snapshots" issue :)
• mdoms 5y ago
  Title is a bit misleading, there's no insider info here. This is all stuff we knew from the official statements, the blog post, reddit and twitter.
• yabones 5y ago
  What's a good Jira replacement? Redmine? Phabricator? OpenProject? Just leaving the jira server alone and hoping there's no new and exciting zero-days? One thing is clear, these guys are a bunch of cowboys who can't be trusted with any amount of data.
  • elxr 5y ago
    If you're hosting your code on GitHub, then GitHub projects is definitely worth using. Does everything I used to use Jira for, but feels more modern and lightweight. Also, it has dark mode.
  • deleted 5y ago
    [deleted]
  • originalvichy 5y ago
    For pure engineering teams it’s either Gitlab or Azure Devops. Those are the most common competitors I hear about. If you have non-engineers the choice gets trickier.
  • dzikimarian 5y ago
    I'm on the same boat. Currently best choice seems to be youtrack, which has reasonable licensing model for self hosted option.
  • 420official 5y ago
    I'm not at all familiar but a tweet linked from the OP and written by the author plugs https://linear.app/ https://linear.app/
    • gkoberger 5y ago
      Linear is phenomenal. Probably built for a different audience than Jira (it's like Superhuman for tickets), but if you want something that works well and is opinionated I highly highly recommend it.
    • bloopernova 5y ago
      Linear has a dark mode. I'm already won over! ;)
• kache_ 5y ago
  return to monke vi your todolists on an ec2 box
• Cederfjard 5y ago
  PSA, because I’m seeing a lot of JIRA in this thread: Since the 2017 rebranding, Jira is no longer officially written in all caps: https://community.atlassian.com/t5/Feedback-Forum-articles/A-new-look-for-Atlassian/ba-p/638077 https://community.atlassian.com/t5/Feedback-Forum-articles/A... (You can argue how successful it was when people are still using the old style in 2022). It also makes more sense, since Jira is not an acronym, it’s a truncation of Gojira, inspired by Bugzilla/Mozilla.
  • spookthesunset 5y ago
    Yeah, I've never typed it as anything but JIRA. Pretty sure my auto-complete will vouch for that.
• ordiel 5y ago
  All I can say as an Attlassian Server products user is that the moment they say it was Cloud or nothing, I choose nothing. I much rather running Gittea on a raspberry pi that I CONTROL than having to have the impotence of doing nothing for more than a week. + having work at cloud companies and having been requested to "collect customer data" to hand it over to the government I would NEVER move critical pieces to anyone else's infa... (Note: I am not supporting crime, but I rather to have privacy and criminals than living on an authoritarian regime where a dictator who knows everything abot everyone keeps "peace".... Yes I am looking at you China!) If mistakes will be made, at least I wont pay others to do them for me....
  • Phelinofist 5y ago
    As I understood it is not "Cloud or Nothing" but "Cloud or Data Center" - is this wrong?
    • yabones 5y ago
      AFAIK the Datacenter pricing starts at 500 users and goes up from there. So a small org could end up paying 5-10x what they were before on the Server license.
    • grnmamba 5y ago
      Unlike server, Data Center starts at 42.000$ per year. For most SMBs, it's cloud or nothing (or a different vendor, of course).
    • callamdelaney 5y ago
      Where do you think the cloud lives?
    • rsstack 5y ago
      The on-prem offering of Atlassian was discontinued. Existing contracts are being honored but as of March 2022, that's the end of the line for it. Maybe it will be revived now.
    • magicalhippo 5y ago
• escot 5y ago
  When doing bulk deletes like this what safe guards do you put in place, other than testing the script up/down in another environment, turning off app servers etc (which Im guessing they did not do)?
  • mh- 5y ago
    Depends how complex the query/procedure is. Naive approach, replace delete with select and see if you're surprised at the results. More mature approach, especially in an environment where engineers are running bulk changes against the database, you don't do bulk deletes. You change that delete into an update that marks things for later collection. One tactic I've seen that worked, assuming you have straightforward relational tables: you add a "marked for deletion" column whose value is an identifier for the single run of the bulk job you just did. Then you can query rows with that value in that column to ensure it had the desired effect. If you're satisfied, you run another bulk job which doesn't re-run your original query.. it just deletes rows with that "marked" value. Lots of places rely on schema-enforced foreign keys and cascading deletes though. In that case, my recommendation is: don't.
  • nijave 5y ago
    Canary deploys i.e. start with a couple customers and do manual validations, wait a little bit of time (maybe a few days) before incrementally rolling it out to larger amounts of customers. It's not clear I'd the issue affected all tenants where the script ran--which it sounds like it did. It wouldn't be as effective if it only effected certain tenants (maybe with a specific config)
• knbrlo 5y ago
  My current employer uses Jira but we seem to have not been affected by this. Hopefully those customers affected are able to press Atlassian for improvements from notification time, backups, usability etc.
• bitwise101 5y ago
  This talk from Atlassian aged well https://conferences.oreilly.com/software-architecture/sa-eu-2019/public/schedule/detail/78930.html https://conferences.oreilly.com/software-architecture/sa-eu-...
  • danuker 5y ago
    I am tired of survivor-biased "best practices" advice. I wonder which practices contained there are the worst practices.
    • galori 5y ago
      This is interesting but can you expand? My understanding of survivor bias is that you're getting a skewed picture because some of the data was excluded completely.
      • throwaway892238 5y ago
        In this case it's survivor bias in that "We did these things and we didn't fail, ergo these things must be great." Whenever you see a talk like this, always assume that it's BS. It might not be used by any real customers, or might still be in development. There might be a bunch of fires happening all the time due to things the talk doesn't mention. And it might be shuttered the next month if it's too expensive, complicated, obscure, or hard to support. These talks should only be considered aspirational sources of ideas, but never taken as a gold-standard battle-tested model, until they tell you how it fails. Only after you know how a system fails and how to respond to it can it be said to be reliable.
      • danuker 5y ago
        Focusing on the practices of successful companies makes you overlook the millions of other companies with the same practices, yet going bankrupt. It is only through understanding what can fail that you can figure out causation. And since Atlassian failed here, the talk might expose some of the failure's causes, or at least cast doubt over the usefulness of the practices presented.
• anshumankmr 5y ago
  I don't get it. JIRA is working for me.
  • vvpan 5y ago
    Article points out that 400 companies were effected.
  • politelemon 5y ago
    A subset of their customers are affected (badly). Enumerate your blessings!
• gunapologist99 5y ago
  Trello seems to still be up?
• RomanPushkin 5y ago
  > I've never seen a product outage last this long Title should be "Inside the longest outage of all time", without "Atlassian" word in it
  • k8sToGo 5y ago
    If I remember correctly, many years ago PSN was down for months.
• Traster 5y ago
  I remember finding out one of the senior managers from my company ended up as head of software at Atlassian. It was at that point I was convinced Atlassian has no idea what the hell they're doing. I think this demonstrates the point nicely.
  • celim307 5y ago
    After this they might have to boomerang back to your company lol
    • Traster 5y ago
      It's funny you say that, because he actually did end up coming back to work for the company, but it was a while ago now. He doesn't even list Atlassian on his Linkedin, he just pretends he never left hte company
  • sngz 5y ago
    I interviewed for them about 8 years ago. The people interviewing me from the recruiter on the phone to the engineers were some of the most incompetent and unprofessional people I have dealt with. They ended up poaching and hiring some equally incompetent engineers from my wife's startup. The outage and what every one is saying in this thread is no surprise to me.
  • tgtweak 5y ago
    Did you fire them or they quit to take the opportunity? Unless it's the former you're just as guilty :)
• 1970-01-01 5y ago
  Wouldn't you love to see the Atlassian internal JIRA epic for this outage?
  • deleted 5y ago
    [deleted]
• captaincaveman 5y ago
  A dumpster fire of a company that has terrible communication with customers outside of outages as well.
• travisgriggs 5y ago
  > Atlassian is a tech company, built by engineers, building products for tech professionals. I am curious if anyone can provide any more insight on this simplification. I've worked at companies like this. Originally a core of motivated creative individuals make a cool product. As the business grows rapidly, Pournelle's (Iron) Law (of Bureaucracy) takes over. For a variety of reasons, the very capable creators depart and are replaced by less motivated/aware individuals who are glad to have a job and easily compelled to do things to the product that probably should not be done. My guess is that while Atlassian may have originally been one of those cool founder places, it has probably morphed into the more incompetent version that comes with scale all too often. But I don't know. Thus my question if anyone can speak to the true current tech capabilities of this company.
• elesbao 5y ago
  In a side note that someone else already made: it is interesting to see that many companies that uses JIRA also uses Slack but the noise/complaint/mentions comparing when Slack is down is way different. I barely saw people complaning.
  • upbeat_general 5y ago
    I don’t believe slack has been down as long? Slack is generally much more critical than JIRA in order to keep working.
    • adhesive_wombat 5y ago
      It's it though? You can hop onto any of a constellation of other IM platforms, FOSS and not fairly quickly for an instant comms channel, even if you're missing the history. Having all your issue tickets missing is something you can't really deal with unless you have a very recent dump, and even then you can't just fire up Bugzilla and get something working without a lot of migration and administrative effort. You can do without JIRA for a week or two as long as managers understand and you all have a good concept of what work needed doing anyway. Then it starts getting dicey unless someone becomes a human JIRA to connect temporary manual bug tracking systems with everyone involved.
      • adamc 5y ago
        We have all sorts of slack channels set up to coordinate activity, so that internal customers can talk to engineers easily, or engineers can engage with each other. If slack goes down, we'd have to work all that out. For many days, it would be a huge drag on the process, slowing down interactions. Other IM platforms wouldn't solve that just by existing. Sure, in principle one could set up such channels elsewhere, but that takes time, and the communication about it takes considerably more time.
        • adhesive_wombat 5y ago
          Sounds like having a fallback pre-defined would be prudent if it's that important and you don't feel you could collectively extemporise something. "If Slack goes down, the plan is to use WhatsApp/Teams/Jeff's Matrix homeserver in his garage until service comes back. A list of group channels will be emailed if that happens." Then if it does go down, you don't have to waste the first day arguing about the plan.
• digital79 5y ago
  Wow this might top the slack outage, who knew a company this large could take this long to resolve the issue
• parentheses 5y ago
  A case for reducing complexity of software. Also, given the recent GitHub incident spree, it's almost debilitating. The entire tech industry takes a hit when companies like these fail at operations.
• mc4ndr3 5y ago
  They never heard of beta testing, rolling updates, infrastructure as code, federation, customer isolation, or Public Relations. What the heck.
• beachy 5y ago
  This is extremely poor for a large SaaS company. A standard RFP question for SaaS should be: - Can you restore data for a single customer, and if so, what is the RTO for that operation? A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival. But for any decent size multi-tenanted SaaS, it's imperative that you have the ability to selectively restore individual customers. The usual approach is to do a full database restore into a separate instance, then run your pre-prepared "restore customer" scripts to extract a single customer's data from there and pump it across your prod instance. In Oracle for example you might use database links to give your restore code access to prod and also the restore instance at the same time. Atlassian - MUST DO BETTER.
  • drsim 5y ago
    Which SaaS platforms provide account-level restores? If you contact them and say "please restore our data to as it was last week" those I know do not offer this.
    • hotpotamus 5y ago
      I actually did this once with Dropbox, though it wasn't a feature they actually published. I clobbered my Dropbox directory accidentally, but I was able to find a script someone wrote to roll it back to a previous point in time and it worked quite well. After that I also took my own snapshots just in case.
      • MapleWalnut 5y ago
        Dropbox support can rollback your Dropbox account to a previous point in time too.
    • boardwaalk 5y ago
      I wouldn’t expect them to advertise such a thing, but the question is “can they recover from their own mistakes” not “can they recover from mine.” I don’t care if this is with an “account-level restore” or whatever; it shouldn’t be my concern.
• kingofpandora 5y ago
  Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying in Las Vegas instead of taking care of business: https://www.linkedin.com/mwlite/feed/hashtag/atlassianteam22 https://www.linkedin.com/mwlite/feed/hashtag/atlassianteam22 Priorities.
  • kazinator 5y ago
    What? The most inexusable thing when engineering mistakes happen is something like: failing to bring back people's dead friends and family members. Not that some customers of some groupware application weren't communicated with for a week.
  • iamtheworstdev 5y ago
    Can confirm. Saw them there while I was on vacation.
    • benreesman 5y ago
      Jesus, if there was ever an example of the internet making the world smaller. When do execs living it up at the fucking Wynn Encore while the house burns down start to not get another job? They’ll keep pulling this shit until it cost money.
      • benreesman 5y ago
        For clarity: I went through a period where some combination of self-indulgence and legitimate life crisis caused me to take my eye off the ball when it mattered. I’m still trying to kickstart a second act years later, because I’m trailer trash and it’s hard work when you’re that.
        • 5y ago
• xyst 5y ago
  Atlassian about to dip over the next few years as firms around the world slowly remove themselves from their ecosystem of products.
  • ekanes 5y ago
    Right. Feels similar in a way to an ongoing conflict elsewhere... There is what happens now, and what happens over the next decade because people have lost fundamental trust in you.
  • tpmx 5y ago
    Their core customers are unfortunately just as dysfunctional and slow-learning. Think Boeing, etc. Witness: https://jobs.boeing.com/job/annapolis-junction/jira-administrator/185/10945541536 https://jobs.boeing.com/job/annapolis-junction/jira-administ...
  • jlawer 5y ago
    Not to mention a CEO who is more interested in activities outside the company like the green energy transition and politics. As an Aussie I always wanted Atlassian to succeed as we have so few tech companies at that scale or larger. Now I view them as another Oracle. Now they innovate little, they keep ratchetting up prices, pushing deployments to cloud where they make more money. Nickel and dime you for what should be core features (SAML Auth?). They aren't coming up with anything new to keep the value in the ecosystem. They buy applications in, spend a little to make some cross integration and then drop down to a slower development Cadence.
    • eddy_chan 5y ago
      I think that Atlassian's core issues stem from the CEO who is more interested in activities outside the company. The lack of vision starts from the top and flows down through the rest of the company. If MCB is so interested in those things, then he should do the right thing which is to retire from Atlassian and go whole-heartedly after those noble causes.
• Tomsilverberry 5y ago
• xiaodai 5y ago
  Comes across as jerk. How can an outsider say things with such certianty?
• hougaard 5y ago
  Always judge companies on how they handle a crysis, not on how they do when everything runs smoothly.
• dynamohk 5y ago
  Rely on a SaaS, complain then don’t have BCP/DR yourself . Huh
  • zmmmmm 5y ago
    yah, it's interesting our org is in the middle of a big debate around BCP/DR where cloud migration is presented as a no-brainer solution that obviates need for DR. Maybe not?
• sgallant 5y ago
  Thinking about all the folks with sites powered by Confluence and this quote I heard from a customer today: “We build to static HTML, deploy to S3 and Cloudfront, and it’s f*ing bomb proof”
• victor9000 5y ago
  Wow would you look at that, a complete Atlassian puff piece got published in the WSJ just hours ago. How peculiar that the biggest active outage in the history of this company is not mentioned once in this "article". I'm left to assume that PR teams can plant whatever they see fit in the WSJ at a moment's notice. I guess that's what passes for journalism these days. https://www.wsj.com/articles/atlassian-puts-easy-to-use-coding-tools-in-hands-of-business-people-11649886533 https://www.wsj.com/articles/atlassian-puts-easy-to-use-codi...
  • taspeotis 5y ago
    > Whatever else changes for CIOs, troubleshooting may never be far behind. Ms. Rao’s remarks came as Atlassian races to restore cloud-based software applications to roughly 400 companies, after a service outage last week caused by a routine maintenance glitch. > As of Wednesday, she said, services were back online for just under half of the companies hit by the outage, which may take up to two weeks to fully repair. Source: https://www.wsj.com/articles/atlassian-puts-easy-to-use-coding-tools-in-hands-of-business-people-11649886533 https://www.wsj.com/articles/atlassian-puts-easy-to-use-codi...
  • deleted 5y ago
    [deleted]
• passerby1 5y ago
  Do anyone seriously consider changing Jira/confluence to some alternative after this? I personally stopped using Jira a couple of years ago in projects I lead.
  • TrevorAustin 5y ago
    Some PMs at my company have been agitating for a switch from Asana to Jira, and now I (as VP Engineering) can guarantee that will never, ever happen.
    • passerby1 5y ago
      What are PMs' arguments for the switch?
• anotherevan 5y ago
  RT @paularambles: Why say Jira’s down when you can say Atlassian shrugged? https://twitter.com/paularambles/status/1514268251349569538 https://twitter.com/paularambles/status/1514268251349569538
• nijave 5y ago
  Very ironic if you have a look at their values... >Open company, no bullshit >Don’t #@!%the customer https://www.atlassian.com/company/values https://www.atlassian.com/company/values
• faddypaddy34 5y ago
  It's almost as if entrusting business critical data to offsite cloud providers isn't the best solution.
• Rantenki 5y ago
  I've repeatedly asked Atlassian if: 1. They can confirm that they have backups of our data (about a thousand stories, substantial confluence, opsgenie history, and three service desks). 2. Will our integrations, configuration, and customizations also be recovered, or will we need to rebuild those once our data is recovered? I have received no response, and no human is even willing to acknowledge those questions. The service desk staff ignore them as if I never asked. Repeatedly. Also, I've been asking around, and haven't been able to find a single story from somebody that can confirm that they were down, who has had their data recovered.
  • mulletbum 5y ago
    I was down, my instance is fully restored right now. 1. They do, every 25 hours, via snapshot. I have spoked to their team since the incident and that same thing is in this article. 2. Yes, they recover all of it. Some things have had issues, external mailboxes attached to service management projects, some attachment rendering slowness. Filters needing to be overlayed into our instance again, but otherwise it is running fine again. Not sure what to tell you other than they are fixing life saving companies first, then the rest. That is what they have told us.
    • Aeolun 5y ago
      Life saving Jira tickets?
      • flarg 5y ago
        FWIW Jira and Confluence and used in life sciences firms, though they are rarely more useful than Google Docs. Self hosting in this context was the norm but that is changing.
    • Rantenki 5y ago
      Thank you for responding here. Even a single successful restoration, anecdotal as it is, makes me feel a lot better about the situation. I was honestly wondering if Atlassian was just pushing the date out to soften the eventual backlash when they had to announce data loss.
• linsomniac 5y ago
  Honest question here: The companies impacted by this, are they not taking backups of their Jira/Confluence/Bitbucket instances? Or is this outage impacting the ability to import those backups? There are some Python scripts that will back up Jira and Confluence. I whipped up a quick script that gets a list of all our bitbucket repos and then it clones those daily as well.
  • dehrmann 5y ago
    A Jira backup blob isn't especially useful. Confluence could be if it's essentially HTML dumps that you could host internally read-only. Bitbucket clearly has backups and a migration path.
    • linsomniac 5y ago
      Why is it not useful? From eyeballing it it looked like the same file I built from our Fogbugz data to import our historic cases into Jira. I'll carve out some time to try doing an import into a new project to see if it loads properly.
      • dehrmann 5y ago
        In this particular case, if you have another account, that could work. What I meant was that Jira isn't very useful unless people can actively use it for issue tracking. It's not all that valuable when it's just a reference.
        • linsomniac 5y ago
          Ooh, I see. In my case I'm thinking more that in the case of Atlassian having lost our data we could reload it from that backup.
• fargle 5y ago
  The lesson learned is that outsourcing at the level of containers or machines and raw compute in the cloud is one thing. It's a pretty fungible open market. But outsourcing one's whole engineering environment to a SaaS on a cloud is just freakin lunacy. Not only do you have things like this outage, but what about simple things like features and versions of the apps changing all the time with no ability to control that. What if they remove or change a feature you use? And expensive vendor-locked-in closed tools have no place in a modern software workflow anyway, on-prem let alone SaaS. Look at the rug-pull for the on-prem Atlasian Server product.
• Cthulhu_ 5y ago
  It kinda reads like their user's data is not separated very cleanly; I've never worked at a SaaS before, but reading this, especially given the size of some customers, I'd want each customer to have their own independent instance, with its own backup pipeline. I was thinking of "just" giving them their own database, but there's been plenty of instances where authentication got botched allowing one user to see another user's data; this should be impossible if things are running on their own instances. Note that I'm pretty naïve and armchair on this subject, I'll see myself out.
• dirtylowprofile 5y ago
  Around 4/5 years ago I saw their Facebook page and was surprised with the bad reviews. And here we are, companies are still using Atlassian.
• scurraorbis 5y ago
  Don't trust cloud providers with your core business functions. I'd go even further and say don't trust the cloud, period. I think the next big thing is going to be moving back on premise or private cloud as more businesses realize this.
• febeling 5y ago
  Reading this piece is kinda boring. As usual, the root cause is a design defect in their backup-restore functionality. And it's at a complexity level any senior developer could have pointed out to be posing a fatal risk to the company. My guess is many people new about the problem inside, but corporate taboos made it impossible to discuss. I'd bet a fortune on this being the case.
  • 2muchcoffeeman 5y ago
    It’s always good to know that these guys with their huge budgets are just doing what I’m doing and aren’t any better at it though.