18 ms·

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that po

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architecture choices that they're trying to hide. Not that I ever thought using their products was a good idea; to be clear about my position... But at this point anyone continuing to rely on them for anything is asking for the suffering they'll get. Signing up for their crap for a vital business function is like offering your tonker to a snapping turtle.

outworlder 5y ago
> OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production This seems to be exactly what they are doing, as described in the article. They don't have automated tools to do this.
JoBrad 5y ago
It sounds like that’s what they are doing, but it’s manual.
omoikane 5y ago
Not being able to selectively restore data for a subset of users might also indicate that users are not fully isolated from each other, which is worrying for technical and nontechnical reasons.
- indymike 5y ago
  There is nothing non-technical that matters. If we start acting like it does, we incredibly poor decisions that in fact have nothing to do with physical reality, and quickly arrive at unworkable technology.
  - omoikane 5y ago
    Non-technical reasons include "legal" and "compliance", which often matters a fair bit. I am not disagreeing that non-technical requirements occasionally lead to poor decisions, for some value of poor.
    - indymike 5y ago
      I live is a state that once tried to legislate that pi = 3.15. The results were tragic, and the attempt to legislate a ratio was a failure, much like systems created by regulation and laws often are. Math is much less forgiving than legal prose. Making database decisions based on criteria that don't make any engineering sense one way or the other is not far off from legislating the value of PI.
VWWHFSfQ 5y ago
Most likely the database tables themselves are just a mixture of everyone's data. There's no true multitenancy. So they have to load the backups into a separate database. Then just go through and individually select/insert into the old database. And then you have to worry about things like foreign key constraints complicating the bulk data loading. Are you going to disable constraint enforcement while you bulk load the data? How does that affect existing and new data from customers using the database? Just a guess. But this sounds like a nightmare honestly.
- radicaldreamer 5y ago
  I can't believe that they would intermix the data in that way... but if they did, godspeed to them, they're likely still overpromising what can be done in this time frame.
  - dabeeeenster 5y ago
    How else do you run a multitenancy platform?
    - radicaldreamer 5y ago
      Sorry, I'm not actually sure... maybe someone who's experienced in backend db can elucidate here. Is it not a good idea to spin up separate db instances for each client/company?
      - nemothekid 5y ago
        I believe you can sign up an account for free or incredibly cheap ($5/user). You would potentially have tens of thousands of databases. Imagine trying to do something like a database migration to add a column. I believe the day to day operations would be a nightmare as no RDBMS has probably had that kind of feature stress tested.
      - some-guy 5y ago
        The company I work at (Workday) does this, but it's for business / liability reasons.
throwaway894345 5y ago
I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks their boxes and it has such a huge list of features that nominally it appears to tick the product development boxes (ignoring more subjective concerns like "quality", "performance", and "usability"). I would really like to try working in an organization that uses something simpler, like Trello (although now that this is also an Atlassian property, maybe not exactly Trello?).
- sam0x17 5y ago
  And yet, it actually is set up in an extremely opinionated annoying way. For example there is no way to actually assign multiple users to the same ticket, which is a big problem if your org legitimately does pair programming (mine does for juniors)
  - Cd00d 5y ago
    Why not just clone the ticket?
  - robertlagrant 5y ago
    Having a single owner for each ticket is not a bad idea. You can see contributors in git.
- RandallBrown 5y ago
  I think people buy JIRA because you can set it up however you want. I've seen it almost as simple as Trello and much more complicated. It doesn't have to be terrible, it just usually is. If JIRA didn't allow you to make it terrible, it wouldn't allow for some of the absurd things that people want it for and those companies might not buy it.
  - a4isms 5y ago
spaetzleesser 5y ago
“horrendous monolithic architecture “ I don’t really understand what this has to do with “monolithic” or not. Atlassian’s software is probably very complex and convoluted but from my experience it’s almost impossible to keep a clean architecture in a software system that has grown over many years and is used and customized by many customers so you have to avoid breaking backwards compatibility.
qiskit 5y ago
> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point How would they lose committed data? Even after restoring the backups can't they run the logs so that everyone is caught up?
- drjasonharrison 5y ago
  Are you assuming that they record the events in a way that can be played back?
- mh- 5y ago
  (There's a tacit assumption here that the data across tenants is commingled in tables, and that's being disputed elsewhere in the thread, but playing along..) You wouldn't be able to do that without forcing downtime for all customers, for the duration it takes to restore the snapshot and then replay the logs. Not to mention the risks of the process failing somehow You could narrow the window to just the "replay" portion, if you were able to stand up an extra database/infra, to switch over to when it was ready. But at some point you'd probably still have to go read only to checkpoint the logs and begin the replay. It's of course possible to do something more complicated here and stream the changes then eventually enact a failover, but this would all be too complex and error prone to introduce in their current crisis mode. It's something I'd suggest considering when architecting their DR/BCP, but it's too late for that kind of elegance (and complexity) now.
ksala_ 5y ago
Personally, given the multi-day outage, I think I would just restore everything to a separate system, and then only point the affected customers to this new system. Take the hit of having two fully separated system initially, and work on reconciling them after without having to worry about the on-going outage. I wonder if they're not doing this due to some tech limitations, to avoid taking the financial cost of running two systems, or to avoid having to reconcile the systems.
- bigtones 5y ago
  That's a really good idea.
- mandevil 5y ago
  At a big multi-tenancy company I used to work at, the problem would have been the accessory machines: we had something like 15-20 different machines around the main DB and API machines, running cron jobs, terminating SSL connections, load balancing, sending alerts to us and customer emails out, etc. And while the backing up and failing over on DB and API machines was a well documented, thoroughly tested process... the other machines were all custom jobs that were very poorly documented, with who knows what scripts running on them, that might or might not be important. Trying to replicate all of that during an emergency would have been a challenge. For just this sort of problem, we actually had three DB servers running all the time: active, passive, and hour behind with the ability to break hour behind's copying of the write-ahead log of active as the DBA's secret weapon for just this problem. If all customers had accidentally lost an hours worth of data it would have been embarrassing, but much less than completely shutting out hundreds of paying customers for two weeks, I think?
underdeserver 5y ago
> Simple concepts aren't that simple at their scale It's true that nothing is simple at scale, but it's important to note that simple concepts are the only concepts that work at scale.
more_corn 5y ago
Yeah, I’m thinking the exact same thing. Perhaps they don’t have the right people on hand to do hard things like this. They also apparently lack an incident response plan since a critical component of that is coms to affected customers. They also lack good practices around preventing human error. It should not have even been possible to make the initial mistake. It certainly should have involved multiple steps of “are you sure” and potentially even review. Sounds like an operations shit show. Glad it’s not my circus.
- robertlagrant 5y ago
  They have great practices; they even published them. They just didn't follow them here.
tus666 5y ago
> OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production You don't think that's exactly what they are doing?

outworlder 5y ago

> OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production This seems to be exactly what they are doing, as described in the article. They don't have automated tools to do this.

JoBrad 5y ago

It sounds like that’s what they are doing, but it’s manual.

omoikane 5y ago

Not being able to selectively restore data for a subset of users might also indicate that users are not fully isolated from each other, which is worrying for technical and nontechnical reasons.

indymike 5y ago
There is nothing non-technical that matters. If we start acting like it does, we incredibly poor decisions that in fact have nothing to do with physical reality, and quickly arrive at unworkable technology.
- omoikane 5y ago
  Non-technical reasons include "legal" and "compliance", which often matters a fair bit. I am not disagreeing that non-technical requirements occasionally lead to poor decisions, for some value of poor.
  - indymike 5y ago
    I live is a state that once tried to legislate that pi = 3.15. The results were tragic, and the attempt to legislate a ratio was a failure, much like systems created by regulation and laws often are. Math is much less forgiving than legal prose. Making database decisions based on criteria that don't make any engineering sense one way or the other is not far off from legislating the value of PI.

VWWHFSfQ 5y ago

Most likely the database tables themselves are just a mixture of everyone's data. There's no true multitenancy. So they have to load the backups into a separate database. Then just go through and individually select/insert into the old database. And then you have to worry about things like foreign key constraints complicating the bulk data loading. Are you going to disable constraint enforcement while you bulk load the data? How does that affect existing and new data from customers using the database? Just a guess. But this sounds like a nightmare honestly.

radicaldreamer 5y ago
I can't believe that they would intermix the data in that way... but if they did, godspeed to them, they're likely still overpromising what can be done in this time frame.
- dabeeeenster 5y ago
  How else do you run a multitenancy platform?
  - radicaldreamer 5y ago
    Sorry, I'm not actually sure... maybe someone who's experienced in backend db can elucidate here. Is it not a good idea to spin up separate db instances for each client/company?
    - nemothekid 5y ago
      I believe you can sign up an account for free or incredibly cheap ($5/user). You would potentially have tens of thousands of databases. Imagine trying to do something like a database migration to add a column. I believe the day to day operations would be a nightmare as no RDBMS has probably had that kind of feature stress tested.
    - some-guy 5y ago
      The company I work at (Workday) does this, but it's for business / liability reasons.

throwaway894345 5y ago

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks their boxes and it has such a huge list of features that nominally it appears to tick the product development boxes (ignoring more subjective concerns like "quality", "performance", and "usability"). I would really like to try working in an organization that uses something simpler, like Trello (although now that this is also an Atlassian property, maybe not exactly Trello?).

sam0x17 5y ago
And yet, it actually is set up in an extremely opinionated annoying way. For example there is no way to actually assign multiple users to the same ticket, which is a big problem if your org legitimately does pair programming (mine does for juniors)
- Cd00d 5y ago
  Why not just clone the ticket?
- robertlagrant 5y ago
  Having a single owner for each ticket is not a bad idea. You can see contributors in git.
RandallBrown 5y ago
I think people buy JIRA because you can set it up however you want. I've seen it almost as simple as Trello and much more complicated. It doesn't have to be terrible, it just usually is. If JIRA didn't allow you to make it terrible, it wouldn't allow for some of the absurd things that people want it for and those companies might not buy it.
- a4isms 5y ago

spaetzleesser 5y ago

“horrendous monolithic architecture “ I don’t really understand what this has to do with “monolithic” or not. Atlassian’s software is probably very complex and convoluted but from my experience it’s almost impossible to keep a clean architecture in a software system that has grown over many years and is used and customized by many customers so you have to avoid breaking backwards compatibility.

qiskit 5y ago

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point How would they lose committed data? Even after restoring the backups can't they run the logs so that everyone is caught up?

drjasonharrison 5y ago
Are you assuming that they record the events in a way that can be played back?
mh- 5y ago
(There's a tacit assumption here that the data across tenants is commingled in tables, and that's being disputed elsewhere in the thread, but playing along..) You wouldn't be able to do that without forcing downtime for all customers, for the duration it takes to restore the snapshot and then replay the logs. Not to mention the risks of the process failing somehow You could narrow the window to just the "replay" portion, if you were able to stand up an extra database/infra, to switch over to when it was ready. But at some point you'd probably still have to go read only to checkpoint the logs and begin the replay. It's of course possible to do something more complicated here and stream the changes then eventually enact a failover, but this would all be too complex and error prone to introduce in their current crisis mode. It's something I'd suggest considering when architecting their DR/BCP, but it's too late for that kind of elegance (and complexity) now.

ksala_ 5y ago

Personally, given the multi-day outage, I think I would just restore everything to a separate system, and then only point the affected customers to this new system. Take the hit of having two fully separated system initially, and work on reconciling them after without having to worry about the on-going outage. I wonder if they're not doing this due to some tech limitations, to avoid taking the financial cost of running two systems, or to avoid having to reconcile the systems.

bigtones 5y ago
That's a really good idea.
mandevil 5y ago
At a big multi-tenancy company I used to work at, the problem would have been the accessory machines: we had something like 15-20 different machines around the main DB and API machines, running cron jobs, terminating SSL connections, load balancing, sending alerts to us and customer emails out, etc. And while the backing up and failing over on DB and API machines was a well documented, thoroughly tested process... the other machines were all custom jobs that were very poorly documented, with who knows what scripts running on them, that might or might not be important. Trying to replicate all of that during an emergency would have been a challenge. For just this sort of problem, we actually had three DB servers running all the time: active, passive, and hour behind with the ability to break hour behind's copying of the write-ahead log of active as the DBA's secret weapon for just this problem. If all customers had accidentally lost an hours worth of data it would have been embarrassing, but much less than completely shutting out hundreds of paying customers for two weeks, I think?

underdeserver 5y ago

> Simple concepts aren't that simple at their scale It's true that nothing is simple at scale, but it's important to note that simple concepts are the only concepts that work at scale.

more_corn 5y ago

Yeah, I’m thinking the exact same thing. Perhaps they don’t have the right people on hand to do hard things like this. They also apparently lack an incident response plan since a critical component of that is coms to affected customers. They also lack good practices around preventing human error. It should not have even been possible to make the initial mistake. It certainly should have involved multiple steps of “are you sure” and potentially even review. Sounds like an operations shit show. Glad it’s not my circus.

robertlagrant 5y ago
They have great practices; they even published them. They just didn't follow them here.

tus666 5y ago

> OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production You don't think that's exactly what they are doing?