4 ms·

Textbook on how not to handle the issue. Deleted Twitter, sanitized Facebook and more than 2 days to admit the issue. There is discussion on twitter that the c

by pgrote 7y ago

Textbook on how not to handle the issue. Deleted Twitter, sanitized Facebook and more than 2 days to admit the issue. There is discussion on twitter that the company said the backups were on the same network as the data. Hopefully there is an offsite backup available. https://twitter.com/ConleyU/status/1151862278909825024 https://twitter.com/ConleyU/status/1151862278909825024 https://twitter.com/MRasconCPA/status/1151894366291734533 https://twitter.com/MRasconCPA/status/1151894366291734533 https://twitter.com/hockeygirlPDX/status/1151945932935585792 https://twitter.com/hockeygirlPDX/status/1151945932935585792 Ouch. This is the sort of stuff that can kill a company. Does Quickbooks with the cloud option offer local backups?

LeifCarrotson 7y ago
As the resident tech guy in a small shop with Quickbooks, yes, you can do local backups. We're not using iNSYNQ for cloud hosting, though.
jacquesm 7y ago
It is interesting how often I see companies that refuse to make back-ups of their cloud hosted data on the assumption that this is now someone else's problem. I also have a - recent - case of a very large manufacturer of storage solutions that managed to fuck up a restore of a raid array to the point that a whole pile of companies lost their data. Backups are so simple, and yet the only times people seem to realize their true value is when they don't have any.
- ygjb 7y ago
  I am not arguing the importance of backups, but tbh, simple is not a fair statement. At the very least, if you have backups, you might have the ability to rebuild/redeploy, but for any non-trivial system, without a comprehensive DR plan that is drilled regularly, even having backups won't save the business if the tooling to get those backups into a production ready state.
- gowld 7y ago
  It's also a major flaw of the cloud and SaaS providers. They don't encourage customers to maintain local (or alternate-cloud) backups, because they refuse to admit the appearance of weakness. If a Cloud/SaaS provider is going to claim to be your hassle-free one-stop shop, shouldn't they provide, as a matter of course, a feature to dump your data daily/weekly/monthly to a designated off-cloud site of your choosing? But they don't, because they want to pretend that they are impenetrable and 100% available.
  - rob-olmos 7y ago
    Would be nice if someone could build a site like https://twofactorauth.org/ https://twofactorauth.org/ for bringing more attention to this alarming and growing issue. The goal to hopefully influence companies into providing such a fundamental ability.
  - londons_explore 7y ago
    A data dump isnt much use if you don't have the software to use said dump. Most SaaS providers don't run open source software...
- PopeDotNinja 7y ago
  > Backups are so simple It gets more interesting when you're trying to backup a distributed database. When your durability strategy is to spread your risk across a lot of nodes in a cluster, and you're federating your cluster across multiple sites, traditional backups may not be a practical option. However, as I write this, I'm realizing that there's nothing simple about distributed anything, and saying "it's hard to..." may not be all that interesting.
  - londons_explore 7y ago
    Pick a data sequence number, and back up each region sperately at the same sequence number. Keep your backups on-site but off-line - you already have many locations. You should practice a location (and it's backups) going offline at the same time as a code bug screws with data in the active system.
  - scurvy 7y ago
    Cassandra handles this pretty nicely. Files are immutable, so files are hard-linked to a snapshot directory. Copy out the snapshot directory, delete it, and things are eventually consistent. We restore our MySQL and Cassandra data every day in our production DC from backups in the DR DC. It's the only way to ensure things are working. It's about 100TB of data, but things zip along nicely now that 100gig networks are (more) common.
    - PopeDotNinja 7y ago
      Neat!
- rectang 7y ago
  > Backups are so simple Disagree. A high-confidence backup plan that includes continuous restoration is of middling difficulty to design. Many get backups wrong and only find out when their archives turn out to be incomplete.
  - deleted 7y ago
    [deleted]
- intricatedetail 7y ago
  Backups are NOT simple, especially when you have things like GDPR in play. I'd imagine companies can't afford to commission backup that is compliant with the regulations. If you spend on a backup system, your company goes down in debt, you don't - there is a chance you go down anyway.
  - jacquesm 7y ago
    If you can not provide a backup system for a fraction of the costs of hosting the production data - including GDPR compliance if that is what is required - then your company does not deserve to be in business to begin with.
- tgsovlerkhgsel 7y ago
  "But we have a contract saying that it can't happen" or "but we have a contract saying it's their problem"... (Which works reasonably well as long as the damages are quantifiable and the other company is actually able to pay them....)
rob-olmos 7y ago
QuickBooks Online does not have a native backup or restore, at all, and it's pretty ridiculous. QBO told me they also can't do on-request restorations with their backups. There's a third-party or two that can use the API to do a backup/restore, but also, still, not all of the QBO data has an API. Eg, IIRC, recurring transaction tasks. Just a matter of time for ransomware to replace data via APIs.
- elliekelly 7y ago
  > Just a matter of time for ransomware to replace data via APIs. I asked about this in a cybersecurity meeting at an investment bank a few years ago - how much security is there around Bloomberg's data feed? All of the algorithms that make million & billion dollar decisions rely on that data so I asked if someone stepped in the middle and fed the algorithms bad data or even slightly delayed data would we even know it had happened? They looked at me like I was a crazy person. I'm actually kind of surprised it hasn't happened yet. Or maybe no one has noticed yet.
  - rob-olmos 7y ago
    That's a fascinating scenario. Even more malicious embedding shellcode in a series of tickers and prices kicked off with a few invalid tickers/prices. I wonder how many other feeds/sources like that in which so many businesses rely on in at least some level of autonomous fashion. Reminds of the story of Google wiping out all of their CDN servers by Diskerase because of a single detected value, IIRC.

by pgrote 7y ago

LeifCarrotson 7y ago

As the resident tech guy in a small shop with Quickbooks, yes, you can do local backups. We're not using iNSYNQ for cloud hosting, though.

jacquesm 7y ago

It is interesting how often I see companies that refuse to make back-ups of their cloud hosted data on the assumption that this is now someone else's problem. I also have a - recent - case of a very large manufacturer of storage solutions that managed to fuck up a restore of a raid array to the point that a whole pile of companies lost their data. Backups are so simple, and yet the only times people seem to realize their true value is when they don't have any.

ygjb 7y ago
I am not arguing the importance of backups, but tbh, simple is not a fair statement. At the very least, if you have backups, you might have the ability to rebuild/redeploy, but for any non-trivial system, without a comprehensive DR plan that is drilled regularly, even having backups won't save the business if the tooling to get those backups into a production ready state.
gowld 7y ago
It's also a major flaw of the cloud and SaaS providers. They don't encourage customers to maintain local (or alternate-cloud) backups, because they refuse to admit the appearance of weakness. If a Cloud/SaaS provider is going to claim to be your hassle-free one-stop shop, shouldn't they provide, as a matter of course, a feature to dump your data daily/weekly/monthly to a designated off-cloud site of your choosing? But they don't, because they want to pretend that they are impenetrable and 100% available.
- rob-olmos 7y ago
  Would be nice if someone could build a site like https://twofactorauth.org/ https://twofactorauth.org/ for bringing more attention to this alarming and growing issue. The goal to hopefully influence companies into providing such a fundamental ability.
- londons_explore 7y ago
  A data dump isnt much use if you don't have the software to use said dump. Most SaaS providers don't run open source software...
PopeDotNinja 7y ago
> Backups are so simple It gets more interesting when you're trying to backup a distributed database. When your durability strategy is to spread your risk across a lot of nodes in a cluster, and you're federating your cluster across multiple sites, traditional backups may not be a practical option. However, as I write this, I'm realizing that there's nothing simple about distributed anything, and saying "it's hard to..." may not be all that interesting.
- londons_explore 7y ago
  Pick a data sequence number, and back up each region sperately at the same sequence number. Keep your backups on-site but off-line - you already have many locations. You should practice a location (and it's backups) going offline at the same time as a code bug screws with data in the active system.
- scurvy 7y ago
  Cassandra handles this pretty nicely. Files are immutable, so files are hard-linked to a snapshot directory. Copy out the snapshot directory, delete it, and things are eventually consistent. We restore our MySQL and Cassandra data every day in our production DC from backups in the DR DC. It's the only way to ensure things are working. It's about 100TB of data, but things zip along nicely now that 100gig networks are (more) common.
  - PopeDotNinja 7y ago
    Neat!
rectang 7y ago
> Backups are so simple Disagree. A high-confidence backup plan that includes continuous restoration is of middling difficulty to design. Many get backups wrong and only find out when their archives turn out to be incomplete.
- deleted 7y ago
  [deleted]
intricatedetail 7y ago
Backups are NOT simple, especially when you have things like GDPR in play. I'd imagine companies can't afford to commission backup that is compliant with the regulations. If you spend on a backup system, your company goes down in debt, you don't - there is a chance you go down anyway.
- jacquesm 7y ago
  If you can not provide a backup system for a fraction of the costs of hosting the production data - including GDPR compliance if that is what is required - then your company does not deserve to be in business to begin with.
tgsovlerkhgsel 7y ago
"But we have a contract saying that it can't happen" or "but we have a contract saying it's their problem"... (Which works reasonably well as long as the damages are quantifiable and the other company is actually able to pay them....)

rob-olmos 7y ago

QuickBooks Online does not have a native backup or restore, at all, and it's pretty ridiculous. QBO told me they also can't do on-request restorations with their backups. There's a third-party or two that can use the API to do a backup/restore, but also, still, not all of the QBO data has an API. Eg, IIRC, recurring transaction tasks. Just a matter of time for ransomware to replace data via APIs.

elliekelly 7y ago
> Just a matter of time for ransomware to replace data via APIs. I asked about this in a cybersecurity meeting at an investment bank a few years ago - how much security is there around Bloomberg's data feed? All of the algorithms that make million & billion dollar decisions rely on that data so I asked if someone stepped in the middle and fed the algorithms bad data or even slightly delayed data would we even know it had happened? They looked at me like I was a crazy person. I'm actually kind of surprised it hasn't happened yet. Or maybe no one has noticed yet.
- rob-olmos 7y ago
  That's a fascinating scenario. Even more malicious embedding shellcode in a series of tickers and prices kicked off with a few invalid tickers/prices. I wonder how many other feeds/sources like that in which so many businesses rely on in at least some level of autonomous fashion. Reminds of the story of Google wiping out all of their CDN servers by Diskerase because of a single detected value, IIRC.