6 ms·

For those of us not up to date, what exactly has happened? Their status page hasn't actually shown why they're having to rebuild.

by lapser 4y ago

haunter 4y ago
>While running a maintenance script, a small number of sites were disabled unintentionally. https://twitter.com/Atlassian/status/1511870509973090304 https://twitter.com/Atlassian/status/1511870509973090304 Most likely they wiped the data
- seanstev 4y ago
  I think sites are their account management stuff. Sounds like they deleted user accounts and not actual data. Notice that only native products are down and not acquisitions. They probably just haven’t migrated those yet.
btgeekboy 4y ago
Supposedly they’re having to basically restore everyone from backups because a system designed to delete old data was a bit more efficient than it should have been: https://reddit.com/r/sysadmin/comments/u14qqq/_/i4a0mk8/?context=1 https://reddit.com/r/sysadmin/comments/u14qqq/_/i4a0mk8/?con...
- xenadu02 4y ago
  Reminder: never delete data for real as your first step. Always mark it deleted along with a time stamp saying when. Then you can hide deleted itemsfrom everything. When a maintenance script goes haywire you can fix the problem quickly. Have a daily job that really deletes records marked deleted after 30 days. If that is too complicated to retrofit then have any mass cleanup script move the records to a CSV file or temporary table. Never ever ever be in a situation where a rogue script or bad SQL WHERE clause means restoring from backups.
  - bpp 4y ago
    Works until you get a bug in the deletion job. I've seen exactly this happen.
    - jdbernard 4y ago
      Yeah, the idea is that by expecting the deletion logic you can make it simpler and more rigorously tested than regularly changing business logic or application code. If you organizationally cannot prioritize quality then nothing can help you.
    - hyperman1 4y ago
      You don't even need a bug. Just a wrong system clock. We had a few windows laptops where something caused them to time travel to 8000 years in the future. Then, they'd slowly spend a few hours deleting every local profile, as nobody had logged in to them for 8000 years. Then, they'd do something to their time zone database and travel back 8000 years. When they started the process, it was unstoppable. Trying to modify the system clock to something sane just caused them to depart to the future again, even if disconnected from the network. None of our users was very amused by this behavior, even if everything important was backed up.
      - tetsusaiga 4y ago
        that is almost cartoonishly nightmarish
        hyperman1 4y ago
        It happened specifically to 1 type of laptop and we only had about 30 of them. So we pulled all of them out of roulation. Then covid struck, so I reformatted most of them with Debian and we gave them away for home schooling. I wonder if I managed to linuxify some kid in the process.
        ghostly_s 4y ago
- jeffybefffy519 4y ago
  Or ransomware got them.
- mdoms 4y ago
  > This data was from a deprecated service that had been moved into the core datastore of our products. That is very interesting. This implies they are backing off, at least somewhat, from their very aggressive microservice strategy. Perhaps they feel like they have gone too far in decomposing their products.

For those of us not up to date, what exactly has happened? Their status page hasn't actually shown why they're having to rebuild.

by lapser 4y ago

For those of us not up to date, what exactly has happened? Their status page hasn't actually shown why they're having to rebuild.

haunter 4y ago

>While running a maintenance script, a small number of sites were disabled unintentionally. https://twitter.com/Atlassian/status/1511870509973090304 https://twitter.com/Atlassian/status/1511870509973090304 Most likely they wiped the data

seanstev 4y ago
I think sites are their account management stuff. Sounds like they deleted user accounts and not actual data. Notice that only native products are down and not acquisitions. They probably just haven’t migrated those yet.

btgeekboy 4y ago

Supposedly they’re having to basically restore everyone from backups because a system designed to delete old data was a bit more efficient than it should have been: https://reddit.com/r/sysadmin/comments/u14qqq/_/i4a0mk8/?context=1 https://reddit.com/r/sysadmin/comments/u14qqq/_/i4a0mk8/?con...

xenadu02 4y ago
Reminder: never delete data for real as your first step. Always mark it deleted along with a time stamp saying when. Then you can hide deleted itemsfrom everything. When a maintenance script goes haywire you can fix the problem quickly. Have a daily job that really deletes records marked deleted after 30 days. If that is too complicated to retrofit then have any mass cleanup script move the records to a CSV file or temporary table. Never ever ever be in a situation where a rogue script or bad SQL WHERE clause means restoring from backups.
- bpp 4y ago
  Works until you get a bug in the deletion job. I've seen exactly this happen.
  - jdbernard 4y ago
    Yeah, the idea is that by expecting the deletion logic you can make it simpler and more rigorously tested than regularly changing business logic or application code. If you organizationally cannot prioritize quality then nothing can help you.
  - hyperman1 4y ago
    You don't even need a bug. Just a wrong system clock. We had a few windows laptops where something caused them to time travel to 8000 years in the future. Then, they'd slowly spend a few hours deleting every local profile, as nobody had logged in to them for 8000 years. Then, they'd do something to their time zone database and travel back 8000 years. When they started the process, it was unstoppable. Trying to modify the system clock to something sane just caused them to depart to the future again, even if disconnected from the network. None of our users was very amused by this behavior, even if everything important was backed up.
    - tetsusaiga 4y ago
      that is almost cartoonishly nightmarish
      - hyperman1 4y ago
        It happened specifically to 1 type of laptop and we only had about 30 of them. So we pulled all of them out of roulation. Then covid struck, so I reformatted most of them with Debian and we gave them away for home schooling. I wonder if I managed to linuxify some kid in the process.
        ghostly_s 4y ago
jeffybefffy519 4y ago
Or ransomware got them.
mdoms 4y ago
> This data was from a deprecated service that had been moved into the core datastore of our products. That is very interesting. This implies they are backing off, at least somewhat, from their very aggressive microservice strategy. Perhaps they feel like they have gone too far in decomposing their products.