5 ms·

Regarding the backup restores: I once worked a company that had a data loss issue. There was nothing else we could do, we had exhausted every option we had ove

Regarding the backup restores: I once worked a company that had a data loss issue. There was nothing else we could do, we had exhausted every option we had over almost 40 hours. At the end of the second day, it was decided to restore from backup. We had done this before, as a test. It took about 12 hours to restore the data and another 12 hours to import the data and get back up and running. One small thing was different this time, and it had huge consequences. As a cost-saving measure, an engineer had changed the location of our backups to the cold-storage tier offered by our cloud provider. All backups, not just 'old' ones. This added 2 additional days to our recovery time, for a total of five days. Interestingly enough, even though we offered a full month's refund to all of our customers, not even half of them took us up on it.

bradknowles 5y ago
Did I ever tell you guys about the time we accidentally nuked all the mailboxes for all the million-plus users on The Global Network Navigator (GNN) site? And how the restore process failed for us? This hasn't been written up at The Register yet, so I don't have a single URL I can share with you.
bombcar 5y ago
In these cases the best thing to do is just give every customer the full month refund; don't make them ask for it.
- sodality2 5y ago
  The best thing to do business-wise, or as a good faith move?
  - rjmunro 5y ago
    What's the difference?
    - treesknees 5y ago
      Good faith would be to lose all of that money to people who are already your customers. Business-wise would be to stay in their good graces and keep those customers by offering the refund, but you don't lose any money to those who either don't care or won't move to a competitor.
      - heisenbit 5y ago
        Reducing the impact analysis within a long running relationship to a single transaction is too narrow. People observe how other people are treated and draw their conclusions even if not impacted. People may tolerate some abuse but it moves them closer to leaving next time. Money lost in the outage may provide for a budget creation to look for an alternative.
        rgj 5y ago
        A lot of people making those decisions don’t care about a refund because it’s other people’s money anyway. In my experience only small companies care about that. Focussing on communicating open and honestly allows them to explain the crap they’re going through because of your mistakes to their bosses, so in fact you can help them save their asses, and they’ll save your ass in return. This is much more important and valuable than a refund. So you should ALWAYS communicate open and honestly, and offer the refund as an option for clients who do not have a boss to account to.
        treesknees 5y ago
        I've seen cases where it was actually _more_ work for a business to process a refund. That money has to go all the way back through accounting/financing, be re-added to budgets for the appropriate groups, etc. It's not something done all the time so it takes extra time for those working on it. It's not like a Visa credit card getting a refund for a wrong coffee order.
      - function_seven 5y ago
        25 years ago the clutch in my beater truck was slipping. I was 16 years old, making $50 a week and had very little in savings. I took that truck to a shop within walking distance of my job. 2 hours later I walked back to see what they found. I figured it would be several hundred dollars for a new clutch, and I'd have to borrow money or something to get it done. I talked to the owner who told be it was an adjustment on the cable. Just needed to be scootched up a bit and it was probably good for another 30k miles. When I asked him how much I owed, he laughed at me and said, "For that? Not worth writing it up. No charge. You want me to show you how to do it yourself next time?" The shop could very easily have charged me 1 hour of labor at their standard rate, maybe $75 or so. Plus a diagnostic or test drive fee. Whatever. He could have told me, "$123.98" and I would have paid it. I wouldn't even have been mad. But I sure as hell wouldn't have remembered the experience so clearly. Nor would I have told a dozen people over the years to take their cars there. And I definitely would not have driven 20 miles out of my way to return to that shop in the future years. Being cynical about this stuff will hurt your brand. It's not obvious. It doesn't show up on the earnings report as a line item. This is service segmentation that seems like a no-brainer to a clueless MBA, but actually matters in the long run. How people view your brand is immensely important. Not forcing customers you already screwed over to then spend more time chasing a refund is not only the right thing to do, it's also good business.
- bzxcvbn 5y ago
  Not every business can afford to go one month without income. What's the best thing for customers? Have the business go bankrupt and irremediably lose access to the service?
  - function_seven 5y ago
    It's 400 clients, not all their user base. They can handle the lost income from a small slice of their customers for one month. And if they can't sustain that, then it's even more imperative that those customers migrate away.
    - enra 5y ago
      Atlassian had almost a billion in free cashflow last year and over a billion in cash. I think they should cover the whole year for these customers.
    - hw 5y ago
      400 clients, but how much of their revenue? Were they all small clients? Not to mention longer tail effects of people moving away to competitors even if they werent directly affected?
    - brongondwana 5y ago
      Fastmail gave 1 month free service to about 2/3 of our customers after a major disk failure that led to about a week of downtime for them as we recovered from backups in... 2005ish I think. Long time ago - it was a pretty major hit and the wave in income is still visible all these years later as a lean month where there's no renewals from that batch! Definitely the right thing to do though.
miketria 5y ago
Hi, I'm Mike and I work in Engineering at Atlassian. Here's our approach to backup and data management: https://www.atlassian.com/trust/security/data-management https://www.atlassian.com/trust/security/data-management - we certainly have the backups and have a restore process that we keep to. However, this incident stressed our ability to do this at scale, which has led to the very long times to restore.
- sizzle 5y ago
  How’s the atmosphere internally Mike? Must be crazy times there. I know this isn’t your fault, so hang in there. Cheers!
  - MuffinFlavored 5y ago
    Whose fault is it? Is it any one person/team’s fault? Management? Culture? “Corporations are people too”
- encryptluks2 5y ago
  You mean your poor practices and bad design. The only way to prevent this type of issue in the future is to admit the failures.
- mdoms 5y ago
  400 tenants doesn't seem like that much scale though...? What will happen if there's an incident affecting more than 0.18% of tenants?
  - mulletbum 5y ago
    It's 400 tenants scattered across all their servers. So they are most likely having to build out servers to pull the data then put it in place. 10x the problem that just restoring a single server would be.
- atlasgone 5y ago
  A friend in Atlassian engineering said the numbers on the trust site are closer to wishful thinking than actual capabilities, and that there has been an engineering wide disaster recovery project running because things were in such bad shape. The recovery part hasn't even started. If Atlassian could actually restore full products in under six hours, they should have been able to restore a second copy of the products exclusively for the impacted customers.
twistedpair 5y ago
This is why I love GCP Cloud Storage. The "colder" tiers are cheaper, and reads simply cost a lot more from there, but they don't slow them down and take days to restore. You pay with dollars, not time for restoring those GCS backups. e.g. Coldline [1] simply has reduced availability in exchange for being cheaper (99.9-99.95% availability, so 43min/mo, way less than "two days"). [1] https://cloud.google.com/storage/docs/storage-classes#coldline https://cloud.google.com/storage/docs/storage-classes#coldli...
- dhzhzjsbevs 5y ago
  Yea, till they rug pull it.

bradknowles 5y ago

Did I ever tell you guys about the time we accidentally nuked all the mailboxes for all the million-plus users on The Global Network Navigator (GNN) site? And how the restore process failed for us? This hasn't been written up at The Register yet, so I don't have a single URL I can share with you.

bombcar 5y ago

In these cases the best thing to do is just give every customer the full month refund; don't make them ask for it.

sodality2 5y ago
The best thing to do business-wise, or as a good faith move?
- rjmunro 5y ago
  What's the difference?
  - treesknees 5y ago
    Good faith would be to lose all of that money to people who are already your customers. Business-wise would be to stay in their good graces and keep those customers by offering the refund, but you don't lose any money to those who either don't care or won't move to a competitor.
    - heisenbit 5y ago
      Reducing the impact analysis within a long running relationship to a single transaction is too narrow. People observe how other people are treated and draw their conclusions even if not impacted. People may tolerate some abuse but it moves them closer to leaving next time. Money lost in the outage may provide for a budget creation to look for an alternative.
      - rgj 5y ago
        A lot of people making those decisions don’t care about a refund because it’s other people’s money anyway. In my experience only small companies care about that. Focussing on communicating open and honestly allows them to explain the crap they’re going through because of your mistakes to their bosses, so in fact you can help them save their asses, and they’ll save your ass in return. This is much more important and valuable than a refund. So you should ALWAYS communicate open and honestly, and offer the refund as an option for clients who do not have a boss to account to.
        treesknees 5y ago
        I've seen cases where it was actually _more_ work for a business to process a refund. That money has to go all the way back through accounting/financing, be re-added to budgets for the appropriate groups, etc. It's not something done all the time so it takes extra time for those working on it. It's not like a Visa credit card getting a refund for a wrong coffee order.
    - function_seven 5y ago
      25 years ago the clutch in my beater truck was slipping. I was 16 years old, making $50 a week and had very little in savings. I took that truck to a shop within walking distance of my job. 2 hours later I walked back to see what they found. I figured it would be several hundred dollars for a new clutch, and I'd have to borrow money or something to get it done. I talked to the owner who told be it was an adjustment on the cable. Just needed to be scootched up a bit and it was probably good for another 30k miles. When I asked him how much I owed, he laughed at me and said, "For that? Not worth writing it up. No charge. You want me to show you how to do it yourself next time?" The shop could very easily have charged me 1 hour of labor at their standard rate, maybe $75 or so. Plus a diagnostic or test drive fee. Whatever. He could have told me, "$123.98" and I would have paid it. I wouldn't even have been mad. But I sure as hell wouldn't have remembered the experience so clearly. Nor would I have told a dozen people over the years to take their cars there. And I definitely would not have driven 20 miles out of my way to return to that shop in the future years. Being cynical about this stuff will hurt your brand. It's not obvious. It doesn't show up on the earnings report as a line item. This is service segmentation that seems like a no-brainer to a clueless MBA, but actually matters in the long run. How people view your brand is immensely important. Not forcing customers you already screwed over to then spend more time chasing a refund is not only the right thing to do, it's also good business.
bzxcvbn 5y ago
Not every business can afford to go one month without income. What's the best thing for customers? Have the business go bankrupt and irremediably lose access to the service?
- function_seven 5y ago
  It's 400 clients, not all their user base. They can handle the lost income from a small slice of their customers for one month. And if they can't sustain that, then it's even more imperative that those customers migrate away.
  - enra 5y ago
    Atlassian had almost a billion in free cashflow last year and over a billion in cash. I think they should cover the whole year for these customers.
  - hw 5y ago
    400 clients, but how much of their revenue? Were they all small clients? Not to mention longer tail effects of people moving away to competitors even if they werent directly affected?
  - brongondwana 5y ago
    Fastmail gave 1 month free service to about 2/3 of our customers after a major disk failure that led to about a week of downtime for them as we recovered from backups in... 2005ish I think. Long time ago - it was a pretty major hit and the wave in income is still visible all these years later as a lean month where there's no renewals from that batch! Definitely the right thing to do though.

miketria 5y ago

Hi, I'm Mike and I work in Engineering at Atlassian. Here's our approach to backup and data management: https://www.atlassian.com/trust/security/data-management https://www.atlassian.com/trust/security/data-management - we certainly have the backups and have a restore process that we keep to. However, this incident stressed our ability to do this at scale, which has led to the very long times to restore.

sizzle 5y ago
How’s the atmosphere internally Mike? Must be crazy times there. I know this isn’t your fault, so hang in there. Cheers!
- MuffinFlavored 5y ago
  Whose fault is it? Is it any one person/team’s fault? Management? Culture? “Corporations are people too”
encryptluks2 5y ago
You mean your poor practices and bad design. The only way to prevent this type of issue in the future is to admit the failures.
mdoms 5y ago
400 tenants doesn't seem like that much scale though...? What will happen if there's an incident affecting more than 0.18% of tenants?
- mulletbum 5y ago
  It's 400 tenants scattered across all their servers. So they are most likely having to build out servers to pull the data then put it in place. 10x the problem that just restoring a single server would be.
atlasgone 5y ago
A friend in Atlassian engineering said the numbers on the trust site are closer to wishful thinking than actual capabilities, and that there has been an engineering wide disaster recovery project running because things were in such bad shape. The recovery part hasn't even started. If Atlassian could actually restore full products in under six hours, they should have been able to restore a second copy of the products exclusively for the impacted customers.

twistedpair 5y ago

This is why I love GCP Cloud Storage. The "colder" tiers are cheaper, and reads simply cost a lot more from there, but they don't slow them down and take days to restore. You pay with dollars, not time for restoring those GCS backups. e.g. Coldline [1] simply has reduced availability in exchange for being cheaper (99.9-99.95% availability, so 43min/mo, way less than "two days"). [1] https://cloud.google.com/storage/docs/storage-classes#coldline https://cloud.google.com/storage/docs/storage-classes#coldli...

dhzhzjsbevs 5y ago
Yea, till they rug pull it.