12 ms·
Incident Report for Confluence
- bkq 5y agoAccording to their Twitter the root cause of this was a maintenance script that was ran https://twitter.com/Atlassian/status/1511870509973090304 https://twitter.com/Atlassian/status/1511870509973090304. Must have been one hell of a maintenance script to knock out this many services, and cause potential data loss.
- gess933 5y agoChange is usually the fault. Maintenance script is just one form of change. All the big boys have had this Cloudflare, Fastly, AWS, GCP, always a change
- msh 5y agoChange is not the cause of a fault. It's almost the same as saying life is the cause of death in a human. Change is life, software that don't change is dead software.
- sinuhe69 5y agoAgree. Bad (or no) change management is the cause, not the change itself.
- andrewclunn 5y agoOr stable software.
- naoqj 5y ago>While running a maintenance script, a small number of sites were disabled unintentionally "sites were disabled", making it sound like they flipped a bit in the database, when they have clearly destroyed a lot of data.
- eps 5y agoMust've been a script they got by email from someone.
- smackeyacky 5y agoWould be funnier if it was code hastily cranked out due to an overdue Jira ticket that some middle manager sprang on an unsuspecting code monkey an hour before the weekly standup. Jira is emotional abuse software.
- stefan_ 5y agoThat's one way to say 70% are still unable to access their data, going on a week now.
- codeptualize 5y agoI feel bad for the people who are dealing with this. Quite stressful to manage a disaster like this, especially when it lasts this long. I hope they get a bonus and some time off once this is resolved.
- gjvc 5y agosave your sympathy for the people who have to use their shitty products
- codeptualize 5y agohaha that as well, plenty of sympathy for everyone, there are no winners here
- dorianmariefr 5y agomaybe competition? time to move to better services
- spicybright 5y agoThere's humans behind every keyboard.
- gjvc 5y agosome of whom are psychopaths and to whom tools like jira are a wet dream for micromanaging people, so your statement is only partially true!
- gjvc 5y agohttps://www.google.com/finance/quote/TEAM:NASDAQ?window=5D https://www.google.com/finance/quote/TEAM:NASDAQ?window=5D
- ShinTakuya 5y agoThe articles I read seem to imply this is less about the outage and more about the current growth numbers being unable to meet the medium term targets that have been painted, and the fact that the P/E ratio is very high.
- glenngillen 5y agoI don’t know the actual numbers involved, but people are already jumping to the conclusion that it means 70% of customers are impacted. “Affected” is doing a lot heavy lifting in this headline. The linked page states that a small number of customers are affects. So it’s 70% of a small number. Where “small number” is defined by Atlassian so we don’t really know the impact here. As best as I can tell my own usage has not been impacted at all.
- throwmeariver1 5y agoEvery HN discussion about this was rather tame and everyone respected the scope of the outtage. The only one jumping to conclusions at the moment is you.
- glenngillen 5y agoI replied precisely because the first comment I saw on this thread was “That's one way to say 70% are still unable to access their data, going on a week now.” I also don’t presume everyone to have read every thread on here every day.
- throwmeariver1 5y agoThe thread and the post are about a partial outtake and 70% of the affected users are not able to access their data. The comment was a snarky jab at the marketing speech. How you would read into it that the commenter meant that 70% of all Atlassian users are affected is not comprehensible.
- rjmunro 5y agoOur Confluence is working fine and as far as I know has been working fine all week - no one has mentioned it in slack or anything, which I would expect. I think Atlassian are probably not helping themselves by calling the number of customers "small" rather than saying the actual number or specifying a percentage of total customers.
- zelphirkalt 5y agoAnother reminder to not trust them with any data you have not stored elsewhere. Next time they might delete your confluence wiki content or whatever else.
- musiccog 5y agoAnother reason to run things locally 'that just work for you' Especially when you are a smaller company/team.
- sschueller 5y agoThe irony is they don't let you host your own system anymore unless you run 500+ accounts. We don't want to and can't switch to a hosted solution so we are still trying to figure out what we are going to do when our licenses expire.
- ZuLuuuuuu 5y agoOur experience is different. We are using Jira cloud for more than 5 years and I don't even remember the last time an outage has affected our work. Meanwhile we still have some self-hosted services (like SVN) which has problems relatively frequently which require manual attention from engineer's time. I probably spent more than a week maintaining our self hosted services last year, compared to 0 minutes for the cloud services we use. Not even including that our self-hosted services are probably not very secure since we sometimes forget to do updates on our servers for a long time.
- zozbot234 5y agoWhy is SVN still in use? You can use a tool like reposurgeon to convert your full repo history to git.
- sschueller 5y agoWhat are "good" self-hosted alternatives to Jira? We are being forced to either go cloud or move to another product. Cloud is not option for us out of legal reasons and looking at this incident it is also not a good idea in general.
- diordiderot 5y agoA list of atlassian product alternatives here https://bye-bye-server.com/ https://bye-bye-server.com/
- jacquesm 5y agoGitlab?
- willis936 5y agoGitlab is a double edged sword. It has bugs, some of them are near untenable (don't you dare try to diff or MR commits with binaries larger than 10 MB). It also has a much smaller feature set than full-blown Jira. I'm not in a scrum setting right now, but if I was I would not want to have to use Gitlab as the one-stop shop.
- XorNot 5y agoIs anyone even using the full Jira featureset? People keep telling me that X doesn't have as many features as Jira, and I'm just genuinely trying to think of what those could possibly be to justify it? Like what does Jira do that Gitlab does not do that's absolutely vital?
- gregdoesit 5y agoThe title of this is post is incorrect. It's not just Confluence that is down, but JIRA and all Atlassian Cloud services for impacted customers. Some customers are also told they should expect to be down for up to another two weeks, making the outage a total of 3 weeks or so [1]. 400 companies are impacted by this outage and at least one of them is a YC company, Bitrise, who are still down, Atlassian is not telling them anything specific, and data loss is a possibility for them [2]. The outage is strange to last this long as it contradicts Atlassian's own admission of how they plan for resillience and their statements of how quickly they can restore data for customers. They specifically say how they utilize multiple DCs, have zero data loss scenarios and their recovery time objective (RTO) is 6 hours for their Tier 1 products (like JIRA). They are failing this objective big time, which questions what is happening behind the scenes. [3] The outage is especially ironic given that Atlassaian have discontinued their self-hosted Server product and are forcing all customers to move to the Atlassian Cloud - this very cloud that is down for many customers. They stopped selling Server licenses in Feb 2021, disallowed changing tiers in Server products this February, and are stopping support for Server products in Feb 2024. This long of an outage for software which companies depend on is mind-boggling. The only similarly long outage I can recall is the 2011 Playstation Network Outage which lasted for 23 days. Good luck with the restoration, which sounds like it's manual work. Hopefully we get a public postmortem from Atlasssian: they are not the company known to publish these, though. [1] https://twitter.com/kjartanmuller/status/1513462616030683138?s=20&t=RUzUTnIEep3A5ZUwB0dETA https://twitter.com/kjartanmuller/status/1513462616030683138... [2] https://twitter.com/gabornadai/status/1513481270411636738?s=20&t=rEneUDLHThM1NozxkFPobg https://twitter.com/gabornadai/status/1513481270411636738?s=... [3] https://www.atlassian.com/trust/security/data-management https://www.atlassian.com/trust/security/data-management
- taspeotis 5y agoYou should include a link to their recovery time objective for their SaaS offerings: 6 hours. https://www.atlassian.com/trust/security/data-management https://www.atlassian.com/trust/security/data-management
- techdragon 5y ago
- mellosouls 5y agoAtlassian successfully restores just over 1 out of 3 affected customers data That's not the title. Quoting from the last update on the page: no reported data loss
- taspeotis 5y ago35% is just over 1 in 3… > we have rebuilt functionality for over 35% of the users who are impacted by the service outage, with no reported data loss The page title is bereft of more information: > Multiple sites showing down/under maintenance
- csomar 5y agoIt's not clear if the "no reported data loss" concerns the 1 of 3 or 3 of 3?
- jacquesm 5y agoI sincerely wonder how bad a cloud outage would have to be for people to wake up to the fact that any cloud solution can turn into 'data gone up into thin air' at the drop of a hat. Even the big ones have had their moments. I'm always painfully aware that these services are only as reliable as the people that run them and the hardware they run on and there is absolutely nothing that will stop a cascade of mistakes or failures to make you feel sorry about not having backups outside of the cloud. And that's before you get into acts of malice. I've seen supposedly very secure and extremely well run orgs lose some or even all of their data to - retrospectively - ridiculously small chances and yet, they happened.
- spicybright 5y agoHonestly, with how investment you need in configuring jira, it's going to take a heck of a lot more than a week long outage to make the cost of a move worth it.
- danuker 5y agoThis is the sunk cost fallacy. Just because you spent a long time on configuring it, it doesn't mean it's worth staying. You should only consider the current situation, and plan for the future.
- lnxg33k1 5y agoWell but it's not that you can lose data in the cloud _ONLY_, you can lose data from everywhere, I think the main difference is who has to work through weekend to restore it, and I am quite happy if that's not me, in my case it would be very unlikely as I would rather quit than work 24/7 to fix a fuckup I didn't cause.. but still
- timthorn 5y agoI think there's a difference in customer attitude, too. Cloud providers are often seen as a silver bullet to resolve reliability issues, but in reality you still need to maintain a BCP.
- case0x 5y agoAs someone who uses Jira and BitBucket at work I wish they were more reliable. Switching to something like Azure DevOps is sadly not an option for us so we’re stuck there.
- taspeotis 5y agoAzure DevOps is a dead end since Microsoft bought GitHub. You can look at ADO’s release notes. The quantity of work released has slowed right down. I mean the output of one three-week sprint (that’s their cadence) was “we are turning some stuff off.” https://docs.microsoft.com/en-us/azure/devops/release-notes/2021/sprint-196-update https://docs.microsoft.com/en-us/azure/devops/release-notes/...
- 9wzYQbTYsAIc 5y agoHere’s their output from a different, recent sprint: https://docs.microsoft.com/en-us/azure/devops/integrate/concepts/rate-limits?view=azure-devops https://docs.microsoft.com/en-us/azure/devops/integrate/conc... Looks like most of their sprints are adding things or fixing things, not removing things.
- taspeotis 5y agoYes mate they did a thing, guess I was wrong. Compare their four most recent sprints: https://docs.microsoft.com/en-us/azure/devops/release-notes/2022/sprint-201-update https://docs.microsoft.com/en-us/azure/devops/release-notes/... https://docs.microsoft.com/en-us/azure/devops/release-notes/2022/sprint-200-update https://docs.microsoft.com/en-us/azure/devops/release-notes/... https://docs.microsoft.com/en-us/azure/devops/release-notes/2022/sprint-199-update https://docs.microsoft.com/en-us/azure/devops/release-notes/... https://docs.microsoft.com/en-us/azure/devops/release-notes/2022/sprint-198-update https://docs.microsoft.com/en-us/azure/devops/release-notes/... To three years ago: https://docs.microsoft.com/en-us/azure/devops/release-notes/2019/sprint-149-update https://docs.microsoft.com/en-us/azure/devops/release-notes/... https://docs.microsoft.com/en-us/azure/devops/release-notes/2019/sprint-148-update https://docs.microsoft.com/en-us/azure/devops/release-notes/... https://docs.microsoft.com/en-us/azure/devops/release-notes/2019/sprint-147-update https://docs.microsoft.com/en-us/azure/devops/release-notes/... https://docs.microsoft.com/en-us/azure/devops/release-notes/2019/sprint-146-update https://docs.microsoft.com/en-us/azure/devops/release-notes/...
- daenney 5y ago35% of affected customers after a 6 day and counting outage. The rest is still in limbo and there’s no real timeline for when it’ll be resolved for those remaining. Atlassian is barely communicating. “Successful” is doing an awful lot of work in the title.
- Intrepidatious 5y agoLuckily (and I count my blessings often for this) our company was not affected by this outage. I can't even fathom the impact it would have on our productivity, communication and general ability to move forward with projects. It definitely does make you reconsider cloud providers for such an integral part of your business.
- dx034 5y agoEspecially as I've never heard from a self hosted Jira instance being down for a week. You just restore the last backup and there you go. Quite simple if it's only one system you have to restore, not millions. To me that shows again that self hosting often beats cloud services. Yes, the vendor knows more about their own product, but delivering services at scale is so much more complex than a small on-prem instance that on prem usually shows higher resiliance. I had the same experience with self hosted Gitlab and Mattermost (as a Slack alternative).
- slifin 5y agoGot a feeling this is going to be a net positive for the world I don't know any developers who are excited about Jira but I know plenty of managers who are
- DocTomoe 5y agoCan we please change the title to reflect that this is an ongoing recovery? The title makes it sound like 2/3 of affected users certain face data loss after a finished recovery process.
- bluedino 5y agoFirst Kronos, now Atlassian...imagine one of the big cloud ERP providers going down for a few weeks. Businesses would be halted across the planet.
- glintik 5y agoIs Atlassin’s CTO still working in a company? If yes, what his daily tasks?
- throwmeariver1 5y agoI guess he is trying to make jokes to lift spirits while sitting behind some senior dev who has to take a deep dive into a legacy codebase no one touched in 5 years.
- YATA1 5y agoVirtue signaling about the topic de jure (BLM, Women at Atlassian, etc.) and how they've sunk hundreds of engineering hours into changing all their repos from "master" to "main" thanks to the heroic work by their Diversity Inclusion and Equity in Engineering team. Bet they wished they spent those hours elsewhere now.
- donatj 5y agoI can’t even imagine what is going on from a technical level. I don’t think we’re Atlassian scale, but we’ve got a couple million users. If you blew up our entire AWS account save our backups we could be nominally back online within an hour or two and fully within 6. I am genuinely puzzled what could be happening with their infrastructure to cause such an outage. Are they self hosted on bare metal and there was a fire or something?
- chrisseaton 5y ago> If you blew up our entire AWS account save our backups we could be nominally back online within an hour or two and fully within 6. Do you regularly exercise this, or is it just a plan?
- donatj 5y agoWe've brought entirely separate instances up a number of times, from scratch. We sell straight to governments and some of them have very strict hosting requirements. Our Chinese installation for instance runs on servers controlled by the Chinese government. I believe they run an AWS compatibility layer from what I've heard, I haven't personally worked on this. We've got a decent amount of experience bringing the whole thing up from scratch, and we build with that in mind. Most of the time in my estimate is for importing the larger detail tables, which we do regularly for our staging environment.
- tyingq 5y agoIt sounds to me like they lost some sort of master reference table that maps customers to services, instances, data, and so on. Which would mean a lot of manual examination to recreate the map. And would give you that situation where maybe the first thirds of restores is somewhat "fast" because it's the bigger customers with more data/logs/activity, and so easier to identify. But now they've hit the long tail. They may also be gated by limitations in-between customers and the backups. Like backups going through a transit VPC, for example. You don't see the bandwidth limitation when you're sending incrementals all day, but try and do full, parallel, multiple customer restores through it, and ... (and perhaps they are afraid of making too many changes on top of the current chaos to fix something like that)
- wonderwonder 5y agoWonder what the issue is where recovery is currently limited to a subset of those affected. Are there multiple issues affecting different customers or is restoration completely manual for some reason and the 2/3 still affected are due to resource (people) constraints? Also side note, I feel bad for the people working those 24/7 shifts. Burnout starts to kick in and more mistakes can happen.
- gumby 5y agoThe situation is appalling but the headline is does not represent the post. The post says that so far they have brought 35% of affected customers back online. The HN title implies that 2/3 of customer data is gone. I can’t really imagine how you would design a system in this day and age that could fail in such a fashion. But anything that discourages the use of Confluence is ok by me.
- throwatl 5y agoThrowaway account, ex-Atlassian employee. This does not suprise me at all. I came by acquisition years ago, and was wondering when something would like this happen. They've deleted internal Slack, internal wiki before. Nobody cares about stability or scalability at Atlassian, their incident process and monitoring is a joke. More than half of the incidents are customer detected. Most of engineering practices at Atlassian focus on only the happy path, almost no one considers what can go wrong. Every system is so interconnected, and there are more SPOF than the employees. So there are million ways this could've happened anyways. Even a minor feature requires 6 months of planning with 3 timezones, while architects and middle-management in Sydney overriding most of the decisions and there is constant reorganization. There is a general sentiment of "We've built JIRA, our stock increased >10x, so everything we say and we will do is true" and all of the other ideas are invalid to them. Also, all other products other than Jira is second-class and not important.
- deleted 5y ago[deleted]
- bin_bash 5y agoIt’s just like JIRA the product. If you gave a team a year to come up with a feature list of the “perfect” project management tool it would probably look like JIRA. It ticks all the feature boxes you’d want. Actually using JIRA on the other hand…
- zelphirkalt 5y agoIt is annoying to use, because every action takes seconds to complete. Nothing in Jira ever feels snappy. Everything feels delayed. Things are hard to find in the jungle of clickable things in the UI. Things seem to be put in places, where I would not intuitively look for them. That is only the UI. There are also bugs lurking in the depth, which will probably never be fixed. Recently I've seen a coworker edit a story description and suddenly the browser showed an API JSON response, with the whole Jira page gone and only that browser native JSON display. Recently I've tried to sort task on my to-do tasks list on the sprint board, but could not arrange them how I wanted, because one of the tasks was a subtask of a task, which was not assigned to me and thus not visible ion my "only my tasks" filter. You have to make all tasks visible, then search for the supertask and then arrange that to be higher in the list, then navigate back to only your own tasks and the subtask will have moved up. Things that take a second with a reasonable UI take 10 or more seconds in Jira. It is a catastrophe of UI design.
- throwaway787544 5y agoRemember, kids: most of the software you have ever used is no longer functional. That browser you used 10 years ago doesn't run today. That server software you used 10 years ago is EOL today. Services come and go. Code dies. Do yourself a favor and plan the sunsetting of every piece of software and service that you use for your business. At the moment you start using it, plan how you will eventually move away from it. It always happens, and planning ahead leads to much fewer headaches. This applies to life in general, too; have an exit strategy.
- rob_c 5y agospeak for industry, in academia there is code still running from 20+ years ago which has been used for publications etc, not a lot but they exist and should be applauded.
- nix23 5y ago>Remember, kids: most of the software you have ever used is no longer functional. That browser you used 10 years ago doesn't run today. That server software you used 10 years ago is EOL today. Services come and go. Code dies. My Multics and MVS Tk4- runs pretty well, my DosBox and Nintendo Emulator too and hell have you checked FreeDOS? >Services come and go Yes >Code dies Don't have to.
- mrits 5y agoThere are very few of us that have to deal with the headaches of our own choices. Even if you are staying at the same company for a decade there is a good chance you moved on to a different team.
- bloopernova 5y agoThis is a great point. I was working with a DevOps team a couple of years ago and they had written some projects to deploy internal web apps to AWS. It came time to remove one of the apps, but the removal date kept slipping. After much hand wringing and meetings, it was discovered that they had no mechanism to uninstall one of their apps. They simply hadn't considered that an app might not be used anymore. Nor did they ever consider needing multiple instances of an app, so all their stuff was using hard coded names. It wasn't a fun project.
- ClickUpCowboy 5y agoHey y'all! Seems like there's a lot of frustration with Jira and Atlassian on this page, and understandably so. This is extremely forward of me, and I hope you'll forgive my professional proactive nature here, but I'm a member of the ClickUp sales team and I'd be happy to learn about your teams and see if we can better serve your needs. If you're interested, or you think your organization could benefit from at least TRYING to see if there's another solution out there, feel free to schedule a meeting here: https://a.clickup.com/c/coreyelder#/select-time https://a.clickup.com/c/coreyelder#/select-time Humbly, ClickUp Cowboy
- djabatt 5y agoI was a CPO of the largest Jira marketplace company for Atlassian. It amazing how much ARR you can drive and it is equally scary how it is to run a SaaS business within the walled garden. Atlassian's DNA is not cloud based software and it will take them time to get there. In the meantime there are so many better solutions for dev groups.