5 ms·

Goodness, I have over 100TB at home and it cost less than a two or three thousand dollars to put in place. That's like $25 per TB. > The stored data amounts to

by bane 1y ago

Goodness, I have over 100TB at home and it cost less than a two or three thousand dollars to put in place. That's like $25 per TB. > The stored data amounts to 858TB (terabytes), equivalent to 449.5 billion A4 sheets. No, the 858TB amounts to under $25k for the government of the 10th largest economy, of one of the most sophisticated countries on the planet, to put in place. Two of those would be less than the price of a new Hyundai Grandeur car. > “It’s daunting as eight years’ worth of work materials have completely disappeared.” So they're clocking in at around 100TB/year or 280GB a day. It's respectable, but not crazy. It's about 12GB/hr, doable with professional, server level hardware with backup moved over dedicated fiber to an offsite location. Multiply the price 10x and you can SSD the entire thing. Even with data sovereignty consideration demanding an entirely 100% home grown solution rather than turning to AWS or Azure, there's no excuse. But it's not like the cloud providers don't already have CSAP certification and local, in country, sovereign clouds [1] with multiple geographic locations in country [2] South Korea is full of granite mountains, maybe its time the government converts one into an offsite, redundant backup vault? 1 - https://erp.today/south-korea-microsoft-azure-first-hyperscaler-to-serve-korean-public-sector/ https://erp.today/south-korea-microsoft-azure-first-hypersca... 2 - https://learn.microsoft.com/en-us/azure/reliability/regions-list https://learn.microsoft.com/en-us/azure/reliability/regions-...

jart 1y ago
The most sophisticated countries and companies are smart enough to use the least sophisticated backup methods. SK needs to backup their data to cassette tapes and tape libraries cost a bit more than that, but not much. Even if they boat their tapes over to an iron mountain in the US, I can't imagine the equipment and service fees are going to cost them more than a few hundred grand. They'll be spending more on the headcount to manage the thing.
- hedora 1y ago
  The operational expenses of this stuff dwarfs the hardware cost. For the tape mountain, you need robots to confirm the tapes still work (mean time to detection of device failure and recovery are key for RAID durability computations). So, someone needs to constantly repair the robots or whatever. If I was being paid to manage that data set, I’d probably find two enterprise storage vendors, and stick two copies of the data set on them, each with primary secondary backup. Enterprise flash has been under a dollar a gigabyte for over a decade, so that’s under $1.7M per copy, amortized over five years. That’s $700K per year, and one of the four copies (at 3-4 sites) could be the primary store. (I can’t be bothered to look up current prices, but moore’s law says there have been six capacity doublings since then, and it still applies to flash and networking, so divide my estimate by 2^6 — so, ten-ish grand per year, with zero full time babysitters required).
  - chii 1y ago
    even with dual vendors, you'd have to still put in place a backup/restore procedures (with the associated software, which may need to be custom). Then you'd need regular testing. These operational concerns will basically double the cost yearly, probably.
    - Maxion 1y ago
      You'll need permanent staff to oversee this, too. This will add at another ~500k+ to your annual expenditure.
samus 1y ago
The article reads like they actually have a fault-tolerant system to store their data. This is probably a data dump for whatever files they are working with that might have started out as a cobbled-together prototype that just picked up momentum and pushed beyond its limitations. Many such cases not only in government IT...
- bane 1y ago
  Looking at the article, my read (which could be wrong) is that the backup was in the same room as the original.
  - lazyasciiart 1y ago
    No. It says that "most systems" in this data center are backed up to separate hardware on a different floor, and then a backup is made at a physically remote location. This particular G-Drive system was not on the standard backup process - it sounds like it was much higher volume than any others, so maybe they couldn't use it. They did have a pilot going to get G-Drive backed up...it was supposed to be scaled up to the whole thing in December.
    - xvector 1y ago
      Wow, rough timing.
    - yrcyk 1y ago
      From what I'm seeing that pilot is about NTOPS (National Total Operation Platform System) which they use to manage everything, and since this was on the floor above the fire it made recovery a lot more complicated There's a high chance i'm missing something though, where did you read about a G-Drive backup system?
      - lazyasciiart 1y ago
        G-Drive is the name of their bespoke file storage system that was taken down by the fire.
WalterBright 1y ago
You can buy a 24Tb drive on sale for $240 or so. Sometimes I wonder why I still try and save disk space :-/
- DaiPlusPlus 1y ago
  Link? Am both curious and skeptical
  - bedstefar 1y ago
    Not trying to account for the parent's claim, but generally check diskprices.com for the latest deals on Amazon (.com, .co.uk, .de, .es, .it etc)
    - WalterBright 1y ago
      Newegg regularly offers sales on hard drives, which is when I buy.
  - import 1y ago
    Very possible with sales and especially non nas grade harddisks
  - joshvm 1y ago
    Seagate Expansion drives are in this price range and can be shucked. They're not enterprise drives meant for constant operation, the big ones are Barracudas or maybe Exos, but for homelab NAS they're very popular.
    - golem14 1y ago
      I have such a NAS for 8 years, (and a smaller netgear one from maybe 16 years ago), and have yet such a disk fail. But you can get unlucky, buying a supposedly new but "refurbished" item via amazon or the seagate store (so I hear), or have the equivalent of the "death star" HDDs, which had a ridicilously high burnout rate (we measured something like > 10% of the drives failed every week across a fairly large deployhment in the field - major bummer. If you use such consumer drives, I strongly suggest to make occasional offsite backups of large mostly static files (movies for most people I guess), and frequent backups of more volatile directories to an offsite place, maybe encrypted in the cloud.
throwaway984393 1y ago
[dead]
Nifty3929 1y ago
~1PB of data, with ingestion at a rate of 12GB per hour, is a tiny amount of data to manage and backup properly for a developed world government. This is silly. Volume clearly should not have been a hinderance. Backup operations are often complex and difficult - but then again it's been worked on for decades and rigorous protocols exist which can and should be adopted. "However, due to the system’s large-capacity, low-performance storage structure, no external backups were maintained" ... "the G-Drive’s structure did not allow for external backups." Clearly [in]competence was the single factor here. This is what happens when you come up with all kind of reasons to do something yourself, which you are not qualified to do, rather than simply paying a vendor to do it for you.
- lloeki 1y ago
  > Backup operations are often complex and difficult It quickly becomes much less so if you satisfy yourself with very crude methods. Sure that would be an imperfect backup in many ways but any imperfect backup is always infinitely better than no backup at all.
Den_VR 1y ago
But it would not have been $25k, it would have been 1-2 million for an “enterprise grade” storage solution from Dell or a competitor. Which isn’t much compared with your granite mountain proposal, nor with the wages of 750,000 civil servants, but it’s a lot more than $25k.
bestham 1y ago
Azure can only be sovereign to the USA.[1] [2] [1]: https://www.computerweekly.com/news/366629871/Microsoft-refuses-to-divulge-data-flows-to-Police-Scotland https://www.computerweekly.com/news/366629871/Microsoft-refu... [2]: https://lcrdc.co.uk/industry-news/microsoft-admits-no-guarantee-of-sovereignty-for-uk-policing-data/ https://lcrdc.co.uk/industry-news/microsoft-admits-no-guaran...

by bane 1y ago

jart 1y ago

The most sophisticated countries and companies are smart enough to use the least sophisticated backup methods. SK needs to backup their data to cassette tapes and tape libraries cost a bit more than that, but not much. Even if they boat their tapes over to an iron mountain in the US, I can't imagine the equipment and service fees are going to cost them more than a few hundred grand. They'll be spending more on the headcount to manage the thing.

hedora 1y ago
The operational expenses of this stuff dwarfs the hardware cost. For the tape mountain, you need robots to confirm the tapes still work (mean time to detection of device failure and recovery are key for RAID durability computations). So, someone needs to constantly repair the robots or whatever. If I was being paid to manage that data set, I’d probably find two enterprise storage vendors, and stick two copies of the data set on them, each with primary secondary backup. Enterprise flash has been under a dollar a gigabyte for over a decade, so that’s under $1.7M per copy, amortized over five years. That’s $700K per year, and one of the four copies (at 3-4 sites) could be the primary store. (I can’t be bothered to look up current prices, but moore’s law says there have been six capacity doublings since then, and it still applies to flash and networking, so divide my estimate by 2^6 — so, ten-ish grand per year, with zero full time babysitters required).
- chii 1y ago
  even with dual vendors, you'd have to still put in place a backup/restore procedures (with the associated software, which may need to be custom). Then you'd need regular testing. These operational concerns will basically double the cost yearly, probably.
  - Maxion 1y ago
    You'll need permanent staff to oversee this, too. This will add at another ~500k+ to your annual expenditure.

samus 1y ago

The article reads like they actually have a fault-tolerant system to store their data. This is probably a data dump for whatever files they are working with that might have started out as a cobbled-together prototype that just picked up momentum and pushed beyond its limitations. Many such cases not only in government IT...

bane 1y ago
Looking at the article, my read (which could be wrong) is that the backup was in the same room as the original.
- lazyasciiart 1y ago
  No. It says that "most systems" in this data center are backed up to separate hardware on a different floor, and then a backup is made at a physically remote location. This particular G-Drive system was not on the standard backup process - it sounds like it was much higher volume than any others, so maybe they couldn't use it. They did have a pilot going to get G-Drive backed up...it was supposed to be scaled up to the whole thing in December.
  - xvector 1y ago
    Wow, rough timing.
  - yrcyk 1y ago
    From what I'm seeing that pilot is about NTOPS (National Total Operation Platform System) which they use to manage everything, and since this was on the floor above the fire it made recovery a lot more complicated There's a high chance i'm missing something though, where did you read about a G-Drive backup system?
    - lazyasciiart 1y ago
      G-Drive is the name of their bespoke file storage system that was taken down by the fire.

WalterBright 1y ago

You can buy a 24Tb drive on sale for $240 or so. Sometimes I wonder why I still try and save disk space :-/

DaiPlusPlus 1y ago
Link? Am both curious and skeptical
- bedstefar 1y ago
  Not trying to account for the parent's claim, but generally check diskprices.com for the latest deals on Amazon (.com, .co.uk, .de, .es, .it etc)
  - WalterBright 1y ago
    Newegg regularly offers sales on hard drives, which is when I buy.
- import 1y ago
  Very possible with sales and especially non nas grade harddisks
- joshvm 1y ago
  Seagate Expansion drives are in this price range and can be shucked. They're not enterprise drives meant for constant operation, the big ones are Barracudas or maybe Exos, but for homelab NAS they're very popular.
  - golem14 1y ago
    I have such a NAS for 8 years, (and a smaller netgear one from maybe 16 years ago), and have yet such a disk fail. But you can get unlucky, buying a supposedly new but "refurbished" item via amazon or the seagate store (so I hear), or have the equivalent of the "death star" HDDs, which had a ridicilously high burnout rate (we measured something like > 10% of the drives failed every week across a fairly large deployhment in the field - major bummer. If you use such consumer drives, I strongly suggest to make occasional offsite backups of large mostly static files (movies for most people I guess), and frequent backups of more volatile directories to an offsite place, maybe encrypted in the cloud.

throwaway984393 1y ago

[dead]

Nifty3929 1y ago

~1PB of data, with ingestion at a rate of 12GB per hour, is a tiny amount of data to manage and backup properly for a developed world government. This is silly. Volume clearly should not have been a hinderance. Backup operations are often complex and difficult - but then again it's been worked on for decades and rigorous protocols exist which can and should be adopted. "However, due to the system’s large-capacity, low-performance storage structure, no external backups were maintained" ... "the G-Drive’s structure did not allow for external backups." Clearly [in]competence was the single factor here. This is what happens when you come up with all kind of reasons to do something yourself, which you are not qualified to do, rather than simply paying a vendor to do it for you.

lloeki 1y ago
> Backup operations are often complex and difficult It quickly becomes much less so if you satisfy yourself with very crude methods. Sure that would be an imperfect backup in many ways but any imperfect backup is always infinitely better than no backup at all.

Den_VR 1y ago

But it would not have been $25k, it would have been 1-2 million for an “enterprise grade” storage solution from Dell or a competitor. Which isn’t much compared with your granite mountain proposal, nor with the wages of 750,000 civil servants, but it’s a lot more than $25k.

bestham 1y ago

Azure can only be sovereign to the USA.[1] [2] [1]: https://www.computerweekly.com/news/366629871/Microsoft-refuses-to-divulge-data-flows-to-Police-Scotland https://www.computerweekly.com/news/366629871/Microsoft-refu... [2]: https://lcrdc.co.uk/industry-news/microsoft-admits-no-guarantee-of-sovereignty-for-uk-policing-data/ https://lcrdc.co.uk/industry-news/microsoft-admits-no-guaran...