16 ms·
Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
- alligatorplum 5y agoMy company is in the middle of a multi year transition from selfhosting atlassian products to using their cloud offerings, and I am sure the infrastructure team/management is very thrilled to see this news.
- croutonwagon 5y agoWhile our tenant was unaffected, i told my management of this issue. They just shrugged and said we could watch and eat popcorn. I was half way expecting them to raise eyebrows I was kinda like....."not really the point of bringing it up." Its worth noting we have had them just delete things within our account before. In fact one of our Senior VP's had their account just....disappear one day. We couldn't @ them in chats, tickets etc. Atlassian just shrugged and "restored" the account and said it was some issue with a stored proc on their backend or something. I have always felt uneasy about how flippant they are in their processes. But it seems that is not shared.
- GeorgeTirebiter 5y agoI'm sure they will also be thrilled waiting sometimes seconds for a character to echo. Good Luck, I suffer with cloud Atlassian every day.
- api 5y agoIt's funny that they recently killed self-hosted Jira. If you'd self-hosted you'd be fine.
- nolok 5y agoTo be fair, Jira screams SaaS. I don't want any of my company trapped on it, but if they were I'm sure as well not going to self host that spawn of hell.
- msla 5y agoSelf-hosting Jira is running a big Java application that has its own directory structure and talks to a Postgres database. It really isn't hard. Even upgrades are automated using shell scripts, although you do need to manually replace some configuration files afterwords for some damn reason. Configuring Jira is complicated, but that's the same regardless of whether you self-host.
- shrdlu2 5y agoSelf-hosted full Atlassian stack (jira, confluence, bamboo, bitbucket) for 6+ years. ~200k tickets. To be honest, it just runs with minimal issues. Mostly downtime is just upgrades. I can remember a few times we've had to add (JVM) memory as our usage increased. Not sure what we're going to do with the discontinuation of server product line. We self-host to keep source code, etc. more than one configuration mistake (or zero-day) away from exposing it to the world.
- briffle 5y agoWe are in the same boat. We are looking at the cloud, but the migration tools are just a pain. I can migrate a project to the cloud from Jira, and it even gives me a report of workflow transitions I have to manualy update/change to fix, which is great. But then, there is no way to keep it in sync. I have to blow that project away in jira cloud, and migrate it again. So I Have to hard-cut over projects, on a system that has dozens and dozens of projects, and somehow have people figure out which ones are where. or one really, really ugly night to cut it all over, and hope it goes well. I'm looking for alternatives, but our team is so invested in some very, very customized workflows, its going to be a pain.
- Nemo_bis 5y agoIs there any way to migrate away from Jira, or is it full lock-in? I mean, is there any migration tool or service to Redmine, Gitlab, MantisBT, Trac or whatever? I think one reason Atlassian was successful is that they always invested a lot of effort in building tools to migrate to their products from any of their competitors (obviously not the other way around).
- media-trivial 5y agoWe're using the cloud version and we're fine too (no outage). What's your point? Are you claiming that self-hosted is never down? Or that self-hosted is more reliable? Because I doubt that. Difference is just that when self-hosted goes down, it doesn't end up in the news.
- rileymat2 5y agoI think there is a strange psychological trait that I have, and others may as well, where I am much more forgiving breaking my own stuff than having someone else do it.
- media-trivial 5y agoPeople are more forgiving torwards themselves and their own folks, I can understand that. I'm just thinking it's important to make decisions based on facts. Some self-hosters walk around with this "my own basement is safer than Amazon datacenters" attitude and that's just not true (in most cases, I guess :D).
- fisf 5y agoNo. The basic assumption is only, that if I pay somebody money for the service and to keep things running, then sudden data loss is unacceptable (and a multi week downtime even more so). Some companies cannot operate effectively without atlassian products, so a fuckup of that scale might just have legal consequences depending on whom it hits.
- vntok 5y ago> Some companies cannot operate effectively without atlassian products Kind of their problem to be frank. > so a fuckup of that scale might just have legal consequences depending on whom it hits. Any contract those companies signed would have a cap on the retributions by Atlassian for trashing SLA targets
- akho 5y ago
- Overtonwindow 5y agoI worked at a company that self hosted Jira, and it was miserable then, I can’t imagine depending on the cloud. I’ll never approve Atlassian products after that experience.
- laurent92 5y agoHow many tickets do you have in your non-Jira ticketing system? I count 100 tickets per person per year.
- thematt 5y agoThey've only killed off one version of self-hosted Jira. Their datacenter edition is still alive.
- throwawayboise 5y ago> If you'd self-hosted you'd be fine. Maybe. But you're counting on your sysadmin(s), who are also managing dozens of other things, to keep up to speed on Jira and its quirks, and apply patches and new versions as they become available without missing any steps or screwing something up. On average, you're still probably better off having a company that knows the product also host it for you, but obviously they can make mistakes too, and the downside is that when they do it might affect all clients, not just one.
- nvr219 5y ago> and the downside is that when they do it might affect all clients, not just one. This is also potentially an upside. For example when us-east-1 went down recently, customers were somewhat understanding because it was "amazon's fault" and everyone was down - it was in the news, etc. If we ran our own data center and that went down, our customers would've just said "why did you morons roll your own data center instead of just using aws?"
- lapser 5y agoFor those of us not up to date, what exactly has happened? Their status page hasn't actually shown why they're having to rebuild.
- haunter 5y ago>While running a maintenance script, a small number of sites were disabled unintentionally. https://twitter.com/Atlassian/status/1511870509973090304 https://twitter.com/Atlassian/status/1511870509973090304 Most likely they wiped the data
- seanstev 5y agoI think sites are their account management stuff. Sounds like they deleted user accounts and not actual data. Notice that only native products are down and not acquisitions. They probably just haven’t migrated those yet.
- btgeekboy 5y agoSupposedly they’re having to basically restore everyone from backups because a system designed to delete old data was a bit more efficient than it should have been: https://reddit.com/r/sysadmin/comments/u14qqq/_/i4a0mk8/?context=1 https://reddit.com/r/sysadmin/comments/u14qqq/_/i4a0mk8/?con...
- xenadu02 5y agoReminder: never delete data for real as your first step. Always mark it deleted along with a time stamp saying when. Then you can hide deleted itemsfrom everything. When a maintenance script goes haywire you can fix the problem quickly. Have a daily job that really deletes records marked deleted after 30 days. If that is too complicated to retrofit then have any mass cleanup script move the records to a CSV file or temporary table. Never ever ever be in a situation where a rogue script or bad SQL WHERE clause means restoring from backups.
- bpp 5y ago
- mattweinberg 5y agoMore discussion here: https://news.ycombinator.com/item?id=30973808 https://news.ycombinator.com/item?id=30973808
- encryptluks2 5y agoI'm honestly surprised their stock hasn't plummeted yet. There is no way I could ever invest in a company that was this terrible.
- Nextgrid 5y agoIf technical competency had any bearing on stock prices they should've been at 0 since long ago. Their stock price is tied to the amount of clueless/shitty companies that will still cling onto their products regardless of what happens, and I don't think this incident is going to change much.
- danuker 5y agoIf the market did not react instantly, it will certainly do so at the next financial report.
- whiteboardr 5y agoHonest question: Since Trello is part of Atlassian aswell - what are good, reliable and above all lightweight alternatives for managing projects without the “pseudo-agile” rabbit holes of functionality?
- maclockard 5y agohttps://linear.app https://linear.app is pretty great
- indigomm 5y agoFree until the end of the year if you get in touch with them - https://twitter.com/karrisaarinen/status/1513574249121812482 https://twitter.com/karrisaarinen/status/1513574249121812482
- atonse 5y agoLinear seems optimized for teams that just build one product at a time. We tried it and while the app was elegant, the concepts just didn't map to our mode of multiple repos and across many different clients (some product, some consulting, etc). I don't know if that changed, but that stopped us from adopting Linear.
- deleted 5y ago[deleted]
- x86_64Ubuntu 5y ago2-weeks is one hell of a Recovery-Time-Objective.
- vntok 5y agoDon't complain, you'll get a few months worth of free credits or something.
- yabones 5y agoWe're mere weeks aware from migrating to their could platform after the self-hosted rugpull. This really doesn't give me confidence in their ability to not break my stuff.
- justin_oaks 5y agoThere are two ways this can go: 1) This outage will get their organization to prioritize work such that it never happens again. 2) This outage is representative of a dysfunctional organization that can't prioritize work correctly. If you've been using Atlassian software for a while and are used to how they prioritize tickets then one of those options seems far more likely than the other.
- laurent92 5y ago> 1) This outage will get their organization to prioritize work such that it never happens again It has already happened in the past.
- uuyi 5y ago2012. https://www.theregister.com/2012/05/09/atlassian_cloud_storage_outage/ https://www.theregister.com/2012/05/09/atlassian_cloud_stora...
- lmc 5y ago"An Atlassian spokesperson contacted by The Reg blamed the loss on the failure of “multiple” storage disks at its cloud storage provider on 28 April." 10 years ago to the month. So as long as everyone plans a vacation for this time in 2032 it'll be OK.
- justin_oaks 5y agoAh, you mean that outages have happened in the past, not that Atlassian has changed because of an outage.
- borplk 5y ago
- Nextgrid 5y agoI bet they screwed up royally, deleted some data and are down to either rebuilding it from logs, caches or other side-effects, or using data recovery software on the storage drives (which might involve third-party companies). I can't see many other reasons why this should take 2 weeks.
- laurent92 5y agoLet me bet: Rebuilding from Jira email notifications. Yes, the diffs in the notifications.
- mirntyfirty 5y agoBut are they gonna be able to accomplish anything without Jira guiding their projects?
- daigoba66 5y agoI've actually done exactly that many years ago for a self-hosted Jira installation that didn't have any backups. You can bet we had backups with regular testing after that.
- deleted 5y ago[deleted]
- debarshri 5y agoI guess they are using JIRA for planning their sprints.
- Rantenki 5y agoAs someone who is impacted, this is obviously immensely frustrating. Worse, outside of "we have rebuilt functionality for over 35% of the users", I haven't seen any reports from the people who have ostensibly been recovered. Next, their published RTO is 6 hours, so obviously they must have done something that completely demolished their ability to use their standard recovery methods: https://www.atlassian.com/trust/security/data-management https://www.atlassian.com/trust/security/data-management Finally, there have been some hints that this is related to the decommissioning of a plugin product Atlassian recently acquired (Insight asset management) which is only really useful to large organizations. I suspect that the "0.18% impacted" number is relative to ALL users of Atlassian, including free/limited accounts, and that the percentage of large/serious organizations who are impacted (and who would have a use for an asset management product), is much higher.
- mey 5y agoDon't have a way to verify this, since you know, but I don't believe we had this plugin installed/enabled, but are still caught in the blast.
- uuyi 5y agoFrom experience: RTO = recovery time objective = lie
- deleted 5y ago[deleted]
- mdeeks 5y agoRTO is so hard to state properly even with regular testing. If someone blows away a critical database, sure you can meet your published RTO. What if we lose 300 of our databases and need to copy snapshots from another region. AWS limits you to 20 concurrent snapshot copies cross region. Which of those databases should you do first? Do you know your entire dependency graph for all 1000 of your services to make the right call? Meanwhile tick-tock your "6 hours" is slipping away. And what if someone nukes our entire AWS account with all of our prod resources? Databases, load balancers, S3 (no such thing as snapshotting there), EC2 instances, etc, etc Those last two are examples are very unlikely but no company is going to say RTO = "probably 6 hours but it could be three weeks if we get ransomwared"
- barnabee 5y agoIf I was made to use Jira I’d be pretty happy about this
- deleted 5y ago[deleted]
- perlgeek 5y agoSo, let me get this straight: * It's been deleted for a week already, they estimate they might need two more weeks. Three in total. * They claim to have "extensive backups", and hundreds of engineers working on it. What? How? This simply doesn't go together. Why would restoring from backup take three weeks? Either their backups aren't complete, or they need new software written for the restore, or something else doesn't add up. I haven't administered their software yet, but what I've learned from the sidelines, at least Jira doesn't seem to be rocket science. A database, an application server (maybe a few instances for larger sites), a bit of config, some caches. This really shouldn't take three weeks to restore.
- Nextgrid 5y agoMy theory in my other comment is that they've deleted some data and are waiting on third-party data recovery specialists. That would explain the timescale.
- deleted 5y ago[deleted]
- jlbooker 5y agoRestore from off-site tape backup. The kind of service where you ship them ~dozen new tapes in a lockbox each week and they ship you the oldest dozen back. It's supposed to be the "if all of a data centers happen to burn to ashes simultaneously" option. If you say "give us all of our tapes, asap" and then have some pour souls swapping them out as fast as the data can be read... it would probably take a few weeks.
- RL_Quine 5y agoTapes are not that slow, 360MB/s per drive, and on a large scale swapping them is completely automatic.
- stonogo 5y agoIn support of your point, 360MB/s is an extremely conservative estimate. I'd expect that from LTO-6, which is around ten years old, and I would certainly hope their backups are on more modern gear than that.
- planb 5y agoAs the person responsible for running Jira and Confluence on premises at my employer I‘m looking forward for the next time one of their sales droids contacts me to make us move to their cloud services (despite me stating that we are not interested multiple times)…
- uuyi 5y agoI would love to listen to that call. There’s nothing that makes me happier than the fearsome squealing noises that enterprise sales drones make when you drop the sales equivalent of a Paveway IV on their pitch. My favourite one was running some software supply chain compliance software on itself and explaining how it was constructed on top of a CVE riddled garbage dump.
- dylan604 5y agoMy favorite experience with a sales rep (not Atlassian related at all) was when I went to the vendor's booth at NAB (huge industry convention for those not familiar). I saw our sales rep who quickly looked down to catch her breath before greeting me. I smiled and let her know that today was her lucky day if she could just introduce me to the tech team she promised I could meet. At one point in my conversation with the tech team, I noticed that a small crowd had gathered around my conversation. I was not intending to hold public court, but I was not going to miss a chance to talk directly in person to these team members. To a non-tech person, it might have been viewed as confrontational. To a tech person to tech person, it was just direct questions being held to the fire for an actual answer vs CSR/Sales rep platitudes.
- senorsmile 5y agoThey are discontinuing on prem after 2022 though. What is your plan if not to migrate to their SaaS offering?
- planb 5y agoWe will probably bite the sour apple (not sure if this is correct English but you’ll get the meaning even if it’s not) and switch to the data center edition which is still on prem but costs approximately twice as much.
- rglover 5y agoI love stuff like this happening, not for the schadenfreude but for the "what not to do" lessons. #3,198,191 Don't automate deletion scripts w/o sufficient recovery options.
- polski-g 5y agoBigger lesson is: do not utilize cloud solutions. Your organization's competency floor is set to the least competent employee of the cloud provider.
- nvr219 5y agoI'd disagree with this lesson. Saying "do not utilize cloud solutions" period is nuts. Google and Microsoft are way better at email hosting and delivery than your on-prem server is unless you spend a ton more money on hardware and engineers to keep it up, which is simply not worth it for many companies. Dropbox is going to have better uptime and lower TCO than your self-hosted owncloud instance. What I will say is it's important for the customer to HAVE THEIR OWN BACKUPS. Don't rely on the vendor - that's the lesson here. If you have all your stuff in AWS back that data up someplace that's not AWS, etc.
- josephcsible 5y agoThere's something humorous when better uptime is cited as a reason to use cloud software instead of on-prem in a thread about a major cloud outage.
- nvr219 5y agoI mean, yes, your point is well taken, but I stand by my point nonetheless.
- Gigachad 5y agoOn prem outages and data loss happens constantly. Much more than cloud hosted issues. They just affect a smaller group each time. It's like how people view the countless car crash deaths as non issue but freak out over a rare train crash.
- polski-g 5y agoHopefully this will make them rethink getting rid of the self-hosted option, as you'd be insane to use their cloud service after this.
- bdefore 5y agoEngineers, do yourself a favor and add to your CV/resume: I will not work for companies that use Atlassian software.
- k8sToGo 5y agoSo like 90% of the companies?
- dylan604 5y agoYeah, it's not really a good piece of advice. If anyone actually reads that bit of information, they'll just shrug and toss the resume in the bin and move to one of the other hundreds of applicants.
- threeseed 5y agoAnd use what else. Their competition is equally bad and offers far less features for larger programs.
- atonse 5y agoYea uhh this kind of blanket statement in a resume would be a red flag to me if I were hiring. It would come across as entitled.
- joepeg 5y ago
- Hamuko 5y agoMy hot take is that this downtime isn't severe enough since my company hasn't noticed it at all, and I can't lobby for an alternative project manager.
- dredmorbius 5y agohttps://nitter.kavin.rocks/kjartanmuller/status/1513462616030683138 https://nitter.kavin.rocks/kjartanmuller/status/151346261603...
- aaronbrethorst 5y agoThe wildest part about this service outage to me is that Atlassian's stock is up 10% over the past month.
- Kwpolska 5y agoBreaking news: stocks are meaningless numbers.
- threeseed 5y agoMight have more to do with them moving their HQ from UK to US. This would allow them access to more investors.
- ben-gy 5y agoFrom what I can find online HQ is in AU, and is staying in AU?
- hogrider 5y agoThe stock market parted with reality sometime during 2020.
- radicalriddler 5y agoThe outage has been in the past week and it's down over 10%... talking about the last months price changes is irrelevant compared to the last week.
- ergocoder 5y agoCan you imagine screwing up this bad and customers still not leave? I'll probably should buy their stock.
- rcurry 5y agoReminds me of the time a group I worked for at a National Lab decided to call the root folder for their project “core”. I bet you can’t guess what filename the backup scripts were configured to ignore…
- outsb 5y agoThis is an expensive lesson I think everyone gets to learn at some point. There's no such thing as a file worth excluding from a backup, it always fucks you, some (like me) more than others. Have to buy twice as much disk? Fine, at least you know you actually have a backup
- lmc 5y agoMakes total sense - the core is the part of the apple everyone throws away.
- ivanvas 5y agoLooking at the company financials and timing, the wooden headquarters might have to wait. I don't envy engineers there right now, some dpts. Stay strong, don't burn out! https://architectureau.com/articles/worlds-tallest-hybrid-timber-tower-to-be-built-in-sydney/ https://architectureau.com/articles/worlds-tallest-hybrid-ti...
- mh- 5y agoWow, those renderings are gorgeous. (If you click the top photo, there's a gallery of 5 more you can scroll through)
- threeseed 5y agoAtlassian Q2 revenue was up 37% y/y. Those sort of growth rates are incredible for a company this size. They can easily afford their timber treehouse.
- syshum 5y agoI bet it does not even blimp their financials Sadly it seems society has assimilated vendors behaving poorly. Data breaches and incompetence does not seem to phase purchasing choices these days
- Throwawayaerlei 5y agoThe "Edifice Complex" has been taken to be a bad portent for a company. Obviously not universal, but if upper management has decided to spend a lot of time focusing on a marque building they're not focusing on the business itself.
- clhodapp 5y agoI wish companies would stop with this "small number of customers" messaging. It always seems disingenuous and, besides, that matters for your internal estimation of business impact but means absolutely nothing to the customers affected.
- tedunangst 5y agoI think they add that because otherwise you read about the problem, panic, and then spend hours digging through your own data to make sure it's all there. Unaffected customers like being told they're not affected.
- seigel 5y agoI am very surprised that there aren't more people/companies announcing "offsite backup capabilities" for JIRA or Confluence, etc.. I did just search for scripts that can do this, I'd probably pay a few bucks for something like that at this point.
- ejb999 5y agoI use JIRA and confluence every single day, I have to, it is everywhere - but imo it is such a horrific toolset in every way (even before this outage), I can't for the life of me figure out how it got so much market-share.
- alkonaut 5y agoBecause all the others are equally bad, basically. The options for enterprisey issue management are quite slim.
- movedx 5y agoI really like ClickUp. Never had an issue.
- alkonaut 5y agoDoes it have good API for extending with what's missing compared to Atlassian/Microsoft/Jetbrains offerings, i.e. the tight connection between issue tracking other aspects than issue tracking such as builds, deployments etc? E.g. how pull requests are related to work items, or which work items end up in a specfic build/deployment etc?
- jamesfinlayson 5y agoIt launched in 2002 - well before my time but I'm guessing that for the first few years of its life there weren't many competitors. Now it has endless competitors and I'm led to believe that it has accumulated lots of features that businesses can't live without but which the average end user never touches.
- Thorentis 5y agoFrom this thread, it sounds like Atlassian needs some better competition ASAP.
- floatinglotus 5y agoAtlassian does not care about individual customers. They are purely driven by numbers. I listened to a presentation by one of their founders a long time ago where he admitted that statistics and number management was part of their DNA. They don’t think about the customer’s name, maybe this is right, maybe not.
- hogrider 5y agoThat's ever single company under capitalism tho. Anything else is only empty platitudes.
- ncmncm 5y agoWith luck, it will never be back up. That would be too good to hope for.
- black_13 5y ago
- hsnewman 5y agoHa, get used to it. The new normal.
- jds375 5y agoWow! I didn’t realize the scope and duration of this outage. This must be doing some serious damage to some of their clients (catastrophic if this does impact JIRA, Confluence, and OpsGenie broadly on a company level). Is there any report of approximately how many (or specific) companies have been affected as a result of this?
- abrookewood 5y agoMeanwhile, people have been waiting since at least 2013 for Atlassian to deliver a way to automate backups for their Cloud offerings: https://jira.atlassian.com/browse/CLOUD-6498 https://jira.atlassian.com/browse/CLOUD-6498 And there's still very little movement: https://community.atlassian.com/t5/Backup-Restore-articles/Early-Access-Program-is-here-Learn-More-amp-Sign-Up/ba-p/1855565 https://community.atlassian.com/t5/Backup-Restore-articles/E... But don't worry! It's in the Cloud! It's all fine!
- ostenning 5y agoWhats the legal implications of such downtime for Atlassian? I could imagine thousands of companies unable to manage employees, product releases, bug fixes, rollbacks and more from this. Could Atlassian be liable for damages?
- deleted 5y ago[deleted]
- johnx123-up 5y agoSomewhat relevant https://news.ycombinator.com/item?id=25590846 https://news.ycombinator.com/item?id=25590846