4 ms·

I can’t even imagine what is going on from a technical level. I don’t think we’re Atlassian scale, but we’ve got a couple million users. If you blew up our ent

by donatj 5y ago

I can’t even imagine what is going on from a technical level. I don’t think we’re Atlassian scale, but we’ve got a couple million users. If you blew up our entire AWS account save our backups we could be nominally back online within an hour or two and fully within 6. I am genuinely puzzled what could be happening with their infrastructure to cause such an outage. Are they self hosted on bare metal and there was a fire or something?

chrisseaton 5y ago
> If you blew up our entire AWS account save our backups we could be nominally back online within an hour or two and fully within 6. Do you regularly exercise this, or is it just a plan?
- donatj 5y ago
  We've brought entirely separate instances up a number of times, from scratch. We sell straight to governments and some of them have very strict hosting requirements. Our Chinese installation for instance runs on servers controlled by the Chinese government. I believe they run an AWS compatibility layer from what I've heard, I haven't personally worked on this. We've got a decent amount of experience bringing the whole thing up from scratch, and we build with that in mind. Most of the time in my estimate is for importing the larger detail tables, which we do regularly for our staging environment.
tyingq 5y ago
It sounds to me like they lost some sort of master reference table that maps customers to services, instances, data, and so on. Which would mean a lot of manual examination to recreate the map. And would give you that situation where maybe the first thirds of restores is somewhat "fast" because it's the bigger customers with more data/logs/activity, and so easier to identify. But now they've hit the long tail. They may also be gated by limitations in-between customers and the backups. Like backups going through a transit VPC, for example. You don't see the bandwidth limitation when you're sending incrementals all day, but try and do full, parallel, multiple customer restores through it, and ... (and perhaps they are afraid of making too many changes on top of the current chaos to fix something like that)
- ollien 5y ago
  > It sounds to me like they lost some sort of master reference table that maps customers to services, instances, data, and so on. What makes you say this?
  - tyingq 5y ago
    Just guessing what would drive a situation where some customers were restored quickly, others stuck for days, and matches somewhat with the limited statements they do make about the issue. Like this one: "The rebuild stage is particularly complex due to several steps that are required to validate sites and verify data." Also matches with it being a cross-product outage. But just a guess.
    - wlonkly 5y ago
      And not just a cross-product outage, but a cross-product outage that includes acquisitions. I doubt there is that much in common between the JIRA/Confluence tenancy model and, say, Statuspage.
toyg 5y ago
Chances are their code is older than yours, hence less automated, and they've since lost some of the people who would know how to do some of the most difficult manual things on failure.
- lamontcg 5y ago
  I would guess outsourcing to contractors is involved in this mess somewhere.
  - notreallyserio 5y ago
    It's also one of those companies built by acquisitions.
contravariant 5y ago
Rebuilding the whole thing might end up being a lot easier than what Atlassians seems to be doing, which is restoring each user individually from a backup (?). Though, going by their speed, this seems to be a weirdly manual process, not sure what is going on there.
technion 5y ago
All I can say here is that my experience with ransomware response is that _every_ company has some document stating they can recover from scratch with backups in x hours, and in no case have I ever seen those hours be accurate, or even close by an order of magnitude.
donmcronald 5y ago
> I can’t even imagine what is going on from a technical level. I always wonder if most SaaS providers are fixated on building massive systems when simple shards would work for most customers. If you compare something like Gitea and GitHub you can probably see the reason. With Gitea your "shard" is siloed away from everyone else because there's no federation (yet?). GitHub is more like a huge social network where everyone can interact. I think the vision for a lot of SaaS is to facilitate B2B collaboration by acting as a proprietary social network rather than via federation between relatively independent shards. The reason is simple IMO; vendor lock-in.
- media-trivial 5y ago
  Former Gitea (Gogs) contributor here. I see your point and you're not really wrong but it's not that easy either: - GitHub has way more features. More features means more scattered data (e.g. different micro-services databases, file storage). - You cannot use Gitea in an organization with hundreds or thousands of engineers. It provides nether the features nor the scalability you need. I like Gitea but I'm happy that my company uses GitHub. - Individuals and open-source use GitHub because it's free and has all the features (including stuff like free CI/CD which is expensive). This is all being paid for by companies like mine who buy expensive enterprise licenses for thousands of engineers. - It's not uncommon to shard by customer/account, even in the cloud and GitHub is actually trying to do this right now for their primary database. Doesn't solve the problem with different storages for different micro-services though.

by donatj 5y ago

chrisseaton 5y ago

> If you blew up our entire AWS account save our backups we could be nominally back online within an hour or two and fully within 6. Do you regularly exercise this, or is it just a plan?

donatj 5y ago
We've brought entirely separate instances up a number of times, from scratch. We sell straight to governments and some of them have very strict hosting requirements. Our Chinese installation for instance runs on servers controlled by the Chinese government. I believe they run an AWS compatibility layer from what I've heard, I haven't personally worked on this. We've got a decent amount of experience bringing the whole thing up from scratch, and we build with that in mind. Most of the time in my estimate is for importing the larger detail tables, which we do regularly for our staging environment.

tyingq 5y ago

It sounds to me like they lost some sort of master reference table that maps customers to services, instances, data, and so on. Which would mean a lot of manual examination to recreate the map. And would give you that situation where maybe the first thirds of restores is somewhat "fast" because it's the bigger customers with more data/logs/activity, and so easier to identify. But now they've hit the long tail. They may also be gated by limitations in-between customers and the backups. Like backups going through a transit VPC, for example. You don't see the bandwidth limitation when you're sending incrementals all day, but try and do full, parallel, multiple customer restores through it, and ... (and perhaps they are afraid of making too many changes on top of the current chaos to fix something like that)

ollien 5y ago
> It sounds to me like they lost some sort of master reference table that maps customers to services, instances, data, and so on. What makes you say this?
- tyingq 5y ago
  Just guessing what would drive a situation where some customers were restored quickly, others stuck for days, and matches somewhat with the limited statements they do make about the issue. Like this one: "The rebuild stage is particularly complex due to several steps that are required to validate sites and verify data." Also matches with it being a cross-product outage. But just a guess.
  - wlonkly 5y ago
    And not just a cross-product outage, but a cross-product outage that includes acquisitions. I doubt there is that much in common between the JIRA/Confluence tenancy model and, say, Statuspage.

toyg 5y ago

Chances are their code is older than yours, hence less automated, and they've since lost some of the people who would know how to do some of the most difficult manual things on failure.

lamontcg 5y ago
I would guess outsourcing to contractors is involved in this mess somewhere.
- notreallyserio 5y ago
  It's also one of those companies built by acquisitions.

contravariant 5y ago

Rebuilding the whole thing might end up being a lot easier than what Atlassians seems to be doing, which is restoring each user individually from a backup (?). Though, going by their speed, this seems to be a weirdly manual process, not sure what is going on there.

technion 5y ago

All I can say here is that my experience with ransomware response is that _every_ company has some document stating they can recover from scratch with backups in x hours, and in no case have I ever seen those hours be accurate, or even close by an order of magnitude.

donmcronald 5y ago

> I can’t even imagine what is going on from a technical level. I always wonder if most SaaS providers are fixated on building massive systems when simple shards would work for most customers. If you compare something like Gitea and GitHub you can probably see the reason. With Gitea your "shard" is siloed away from everyone else because there's no federation (yet?). GitHub is more like a huge social network where everyone can interact. I think the vision for a lot of SaaS is to facilitate B2B collaboration by acting as a proprietary social network rather than via federation between relatively independent shards. The reason is simple IMO; vendor lock-in.

media-trivial 5y ago
Former Gitea (Gogs) contributor here. I see your point and you're not really wrong but it's not that easy either: - GitHub has way more features. More features means more scattered data (e.g. different micro-services databases, file storage). - You cannot use Gitea in an organization with hundreds or thousands of engineers. It provides nether the features nor the scalability you need. I like Gitea but I'm happy that my company uses GitHub. - Individuals and open-source use GitHub because it's free and has all the features (including stuff like free CI/CD which is expensive). This is all being paid for by companies like mine who buy expensive enterprise licenses for thousands of engineers. - It's not uncommon to shard by customer/account, even in the cloud and GitHub is actually trying to do this right now for their primary database. Doesn't solve the problem with different storages for different micro-services though.