4 ms·

I bet they screwed up royally, deleted some data and are down to either rebuilding it from logs, caches or other side-effects, or using data recovery software o

I bet they screwed up royally, deleted some data and are down to either rebuilding it from logs, caches or other side-effects, or using data recovery software on the storage drives (which might involve third-party companies). I can't see many other reasons why this should take 2 weeks.

laurent92 5y ago
Let me bet: Rebuilding from Jira email notifications. Yes, the diffs in the notifications.
- mirntyfirty 5y ago
  But are they gonna be able to accomplish anything without Jira guiding their projects?
- daigoba66 5y ago
  I've actually done exactly that many years ago for a self-hosted Jira installation that didn't have any backups. You can bet we had backups with regular testing after that.
  - uuyi 5y ago
    I would have just quit if I had that landed on me.
    - Aachen 5y ago
      Honestly it doesn't sound too difficult and like a challenge to script something fun. To me. If you ever find yourself in that situation, email is in my profile and we'll work something out :)
      - PyWoody 5y ago
      - PyWoody 5y ago
        Whoops, looks like I commented on the wrong chain.
- BFLpL0QNek 5y ago
  Not much on this on the internet. Gray•Duffy, LLP Settles Two Massive International Data Loss Claims Arising From Computer Server Failures Arc Touch, Inc. vs. Atlassian PTY, Ltd http://m.grayduffylaw.com/?url=https%3A%2F%2Fgrayduffylaw.com%2F2014%2F05%2Fgray%25E2%2580%25A2duffy-settles-two-massive-international-data-loss-claims-arising-from-computer-server-failures%2F&utm_referrer=#2843 http://m.grayduffylaw.com/?url=https%3A%2F%2Fgrayduffylaw.co... https://www.theregister.com/2012/05/09/atlassian_cloud_storage_outage/ https://www.theregister.com/2012/05/09/atlassian_cloud_stora...
- davorak 5y ago
  > Yes, the diffs in the notifications. In my limited experience these diffs can be missing information. I recently had to reconstruct an issue description using these email diffs after two people where editing the description at the same time and it was not 100% accurate, several lines were missing. Going to the 'history' tab on the issue I was able to get the missing lines however, if all you have are emails though you might be out of luck.
- escot 5y ago
  Its like a reverse event sourced system
mrits 5y ago
A few years ago I depended on Vertica (snowflae/clickhouse) type database. When a node went down there was no hope of ever coming back up unless you shut the other nodes down as well. While this was going down of course none of our ingress data was being inserted so it built a queue. When we turned things back on the queue would overload vertica again and we had to repeat the whole thing. Fortunately for us we only stored analytic type data on vertica where customers usually were only interested in the last few hours anyways. So we ended up deleting all historical data and just reprocessing it over months while occasionally prioritizing customers that complained.
jrib 5y ago
this is a scary thought. I need to start being more aggressive about backing things up that are "in the cloud."
- syshum 5y ago
  The cloud is just someone else's servers. 3-2-1-0 Applies to all data, at all time, in all places
  - thisisjasononhn 5y ago
    Normally it's 3-2-1, what does the 0 represent here? 3 copies 2 different formats (ex. HDD, cloud) 1 offsite 0 lost data? :D
    - phone8675309 5y ago
      0 data leaving the system unencrypted would be my formulation
    - nix23 5y ago
      >2 different formats (ex. HDD, cloud) That's not what "format" means, it's more like DB-Dump and DB-VM-Dump, or pure Files and VM-Dump, or something like restic-repo and rsync(pure files).
      - syshum 5y ago
        2 Media, but the HDD / Cloud is correct The idea is to not have the backups stored on the same hardware or even same type of hardware. Same hardware is obvious but same type of hardware is listed because if a manufacturing defect or a known vulnerability is present it would make all of your backups at risk. So you want to have backups stored on 2 desperate types of storage media. HDD and Tape, or Cloud etc...
        nix23 5y ago
        for me it was always 2 different formats (db-dump, vm-dump) because i trust (good) storage-media more then backup-software, for example old veeam-backups cannot be restored with new version, old veeam-software runs not on new esxi etc...
x0x0 5y ago
Everyone who's used jira understands that they don't hire good engineers. If they did, it wouldn't be So. Damn. Slow. They're almost certainly rebuilding something from scratch. If it were an AWS system limitation, almost all of those can be lifted if you ask nicely and are a big account.

laurent92 5y ago

Let me bet: Rebuilding from Jira email notifications. Yes, the diffs in the notifications.

mirntyfirty 5y ago
But are they gonna be able to accomplish anything without Jira guiding their projects?
daigoba66 5y ago
I've actually done exactly that many years ago for a self-hosted Jira installation that didn't have any backups. You can bet we had backups with regular testing after that.
- uuyi 5y ago
  I would have just quit if I had that landed on me.
  - Aachen 5y ago
    Honestly it doesn't sound too difficult and like a challenge to script something fun. To me. If you ever find yourself in that situation, email is in my profile and we'll work something out :)
    - PyWoody 5y ago
    - PyWoody 5y ago
      Whoops, looks like I commented on the wrong chain.
BFLpL0QNek 5y ago
Not much on this on the internet. Gray•Duffy, LLP Settles Two Massive International Data Loss Claims Arising From Computer Server Failures Arc Touch, Inc. vs. Atlassian PTY, Ltd http://m.grayduffylaw.com/?url=https%3A%2F%2Fgrayduffylaw.com%2F2014%2F05%2Fgray%25E2%2580%25A2duffy-settles-two-massive-international-data-loss-claims-arising-from-computer-server-failures%2F&utm_referrer=#2843 http://m.grayduffylaw.com/?url=https%3A%2F%2Fgrayduffylaw.co... https://www.theregister.com/2012/05/09/atlassian_cloud_storage_outage/ https://www.theregister.com/2012/05/09/atlassian_cloud_stora...
davorak 5y ago
> Yes, the diffs in the notifications. In my limited experience these diffs can be missing information. I recently had to reconstruct an issue description using these email diffs after two people where editing the description at the same time and it was not 100% accurate, several lines were missing. Going to the 'history' tab on the issue I was able to get the missing lines however, if all you have are emails though you might be out of luck.
escot 5y ago
Its like a reverse event sourced system

mrits 5y ago

A few years ago I depended on Vertica (snowflae/clickhouse) type database. When a node went down there was no hope of ever coming back up unless you shut the other nodes down as well. While this was going down of course none of our ingress data was being inserted so it built a queue. When we turned things back on the queue would overload vertica again and we had to repeat the whole thing. Fortunately for us we only stored analytic type data on vertica where customers usually were only interested in the last few hours anyways. So we ended up deleting all historical data and just reprocessing it over months while occasionally prioritizing customers that complained.

jrib 5y ago

this is a scary thought. I need to start being more aggressive about backing things up that are "in the cloud."

syshum 5y ago
The cloud is just someone else's servers. 3-2-1-0 Applies to all data, at all time, in all places
- thisisjasononhn 5y ago
  Normally it's 3-2-1, what does the 0 represent here? 3 copies 2 different formats (ex. HDD, cloud) 1 offsite 0 lost data? :D
  - phone8675309 5y ago
    0 data leaving the system unencrypted would be my formulation
  - nix23 5y ago
    >2 different formats (ex. HDD, cloud) That's not what "format" means, it's more like DB-Dump and DB-VM-Dump, or pure Files and VM-Dump, or something like restic-repo and rsync(pure files).
    - syshum 5y ago
      2 Media, but the HDD / Cloud is correct The idea is to not have the backups stored on the same hardware or even same type of hardware. Same hardware is obvious but same type of hardware is listed because if a manufacturing defect or a known vulnerability is present it would make all of your backups at risk. So you want to have backups stored on 2 desperate types of storage media. HDD and Tape, or Cloud etc...
      - nix23 5y ago
        for me it was always 2 different formats (db-dump, vm-dump) because i trust (good) storage-media more then backup-software, for example old veeam-backups cannot be restored with new version, old veeam-software runs not on new esxi etc...

x0x0 5y ago

Everyone who's used jira understands that they don't hire good engineers. If they did, it wouldn't be So. Damn. Slow. They're almost certainly rebuilding something from scratch. If it were an AWS system limitation, almost all of those can be lifted if you ask nicely and are a big account.