5 ms·

If it takes you 6 months to get your software back up and running, is it really a good backup solution ?

toomuchtodo 4y ago
When you pay the hacker with your tax base funds, there is no accountability. Incentives matter, and the county was incompetent. > “The only data that we had that wasn’t backed up was in our auditor's office, where we have our financial software,” Ramos said. Seems like it should’ve been a priority. (a component of my work in risk mgmt is business continuity and disaster recovery, excuse my saltiness)
- verandaguy 4y ago
  I empathize with the instinct to point out a correct solution in a space you specialize in, but as another commenter pointed out, this is a county of about 5000 adults, with a likely very small government. It's very possible that they were under-resourced for this kind of incident, and under-resourced to have a rock-solid backup solution in place. Given the usual IT chops of a small local government, I give them points for having backups at all. Funding from higher levels of government (either state/provincial or federal depending on the region) earmarked for anti-ransomware initiatives, could help mitigate this sort of disaster in the future.
  - deleted 4y ago
    [deleted]
  - verve_rat 4y ago
    It seems that state level governments should be providing the setup needed to run a county. These small places have no hope of operating a government software stack on their own.
    - verandaguy 4y ago
      Yup, that's my current opinion. Counties can be extremely well-funded if they encompass a city, and may have almost no funding at all if they're rural. The needs of each county government are obviously different as well, but there's still a bunch of fixed costs that take up a disproportionately large part of smaller counties' budgets.
  - WalterBright 4y ago
    > under-resourced A 16T drive costs around $400. Shouldn't be so expensive to copy the contents of the disk to it now and then. No special software is needed, either.
    - verandaguy 4y ago
      There's more to backing up public and government records than a big hard drive and someone who can set up rsync. Because they're government records, there's the very good chance that there are compliance/certification requirements, plus there's infrastructure you have to put in to make it auditable while mitigating the risk of a ransomware attack affecting backups, too. Setting up total, compliant, and working backups for a small county can reasonably require a small headcount and some specialized long-term planning. E: that's not to mention that in some lower-population, lower-income counties, $400 isn't trivial to spend. There are definitely places in the States where deciding whether to upgrade a single computer from 2006 for $800 is a long conversation.
      - WalterBright 4y ago
        How is doing nothing being compliant? As for spending $800, how is spending $37,000 more feasible? I find these sorts of exchanges astonishing. A 1Tb drive can be bought for $30 on ebay. Backing up to it is a simple xcopy or rsync command. Then put the drive in the safe. It's as simple as that.
        verandaguy 4y ago
        Ok, how often do you take it out of the safe to do periodic backups? How's access to the safe managed? What happens if you get a vindictive (ex-)employee with access to it? If the person with access to the safe loses the key, or gets in an accident and can't open it anymore, what do you do? How do you audit access to the safe to make sure nobody's pulling down data and selling it for profit on the side? How do you make sure you don't lose all your backups by having a butterfingers IT person drop the drive? How sure are you that the latest backup doesn't contain files hit by ransomware, or worse yet, the ransomware itself? If you're storing financial or medical data about the county's citizens, how are you going to handle the possibility of being legally challenged down the line about data storage practices? Handling any data in an org of almost any size is surprisingly complicated, especially if it's other peoples' data. Most people don't realize the scale of the issue because they're coming from a place of mostly working with at-home backups. In terms of spending feasibility: really depends on how funding gets allocated by the state. Like I said in another comment, the solution would be for the state to subsidize the fixed costs of putting this kind of infrastructure in place.

When you pay the hacker with your tax base funds, there is no accountability. Incentives matter, and the county was incompetent. > “The only data that we had that wasn’t backed up was in our auditor's office, where we have our financial software,” Ramos said. Seems like it should’ve been a priority. (a component of my work in risk mgmt is business continuity and disaster recovery, excuse my saltiness)

verandaguy 4y ago
I empathize with the instinct to point out a correct solution in a space you specialize in, but as another commenter pointed out, this is a county of about 5000 adults, with a likely very small government. It's very possible that they were under-resourced for this kind of incident, and under-resourced to have a rock-solid backup solution in place. Given the usual IT chops of a small local government, I give them points for having backups at all. Funding from higher levels of government (either state/provincial or federal depending on the region) earmarked for anti-ransomware initiatives, could help mitigate this sort of disaster in the future.
- deleted 4y ago
  [deleted]
- verve_rat 4y ago
  It seems that state level governments should be providing the setup needed to run a county. These small places have no hope of operating a government software stack on their own.
  - verandaguy 4y ago
    Yup, that's my current opinion. Counties can be extremely well-funded if they encompass a city, and may have almost no funding at all if they're rural. The needs of each county government are obviously different as well, but there's still a bunch of fixed costs that take up a disproportionately large part of smaller counties' budgets.
- WalterBright 4y ago
  > under-resourced A 16T drive costs around $400. Shouldn't be so expensive to copy the contents of the disk to it now and then. No special software is needed, either.
  - verandaguy 4y ago
    There's more to backing up public and government records than a big hard drive and someone who can set up rsync. Because they're government records, there's the very good chance that there are compliance/certification requirements, plus there's infrastructure you have to put in to make it auditable while mitigating the risk of a ransomware attack affecting backups, too. Setting up total, compliant, and working backups for a small county can reasonably require a small headcount and some specialized long-term planning. E: that's not to mention that in some lower-population, lower-income counties, $400 isn't trivial to spend. There are definitely places in the States where deciding whether to upgrade a single computer from 2006 for $800 is a long conversation.
    - WalterBright 4y ago
      How is doing nothing being compliant? As for spending $800, how is spending $37,000 more feasible? I find these sorts of exchanges astonishing. A 1Tb drive can be bought for $30 on ebay. Backing up to it is a simple xcopy or rsync command. Then put the drive in the safe. It's as simple as that.
      - verandaguy 4y ago
        Ok, how often do you take it out of the safe to do periodic backups? How's access to the safe managed? What happens if you get a vindictive (ex-)employee with access to it? If the person with access to the safe loses the key, or gets in an accident and can't open it anymore, what do you do? How do you audit access to the safe to make sure nobody's pulling down data and selling it for profit on the side? How do you make sure you don't lose all your backups by having a butterfingers IT person drop the drive? How sure are you that the latest backup doesn't contain files hit by ransomware, or worse yet, the ransomware itself? If you're storing financial or medical data about the county's citizens, how are you going to handle the possibility of being legally challenged down the line about data storage practices? Handling any data in an org of almost any size is surprisingly complicated, especially if it's other peoples' data. Most people don't realize the scale of the issue because they're coming from a place of mostly working with at-home backups. In terms of spending feasibility: really depends on how funding gets allocated by the state. Like I said in another comment, the solution would be for the state to subsidize the fixed costs of putting this kind of infrastructure in place.