13 ms·
The government official who insisted that commercial AWS/GCP/Azure couldn't possibly be trusted with keeping the information will be keeping their head low for
by kristianc 1y ago
The government official who insisted that commercial AWS/GCP/Azure couldn't possibly be trusted with keeping the information will be keeping their head low for a few days then...
"The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for external backups."
This is absolutely wild.
- zwnow 1y agoRightfully did not trust these companies. Sure what happened is a disaster for them, but you cant simply trust Amazon & Microsoft.
- oceansky 1y agoFor sure the only error here is zero redundancy.
- kingnothing 1y agoWhy not? You can easily encrypt your data before sending it for storage on on S3, for example.
- zhouzhao 1y agoYou can encrypt them at rest, but data that lies encrypted and is never touched, is useless data. You need to decrypt them as well. Also, plenty of incompetent devops around, and writing a decryption toolchain can be difficult.
- kspacewalk2 1y agoAm I missing something? If you ever need to use this data, obviously you transfer it back to your premises and then decrypt it. Whether it's stored at Amazon or North Korean Government Cloud makes no difference whatsoever if you encrypt before and decrypt after transfer.
- DarkmSparks 1y agoEncryption only protects data for an unknown period of time, not indefinately.
- mikehotel 1y agoIf your threat model includes the TLA types, then backup to a physical server you control in a location geographically isolated from your main location. Or to a local set of drives that you physically rotate to remote locations.
- oceansky 1y agoThey can take the data hostage, the foreign nation would have no recourse.
- Imustaskforhelp 1y agoHave it in multiple countries with multiple providers if money isn't a concern. And are we forgetting that they can literally have a multi cloud backup setup in their own country as well or incentivize companies to build their datacenters there in partnership with them of sorts with a multi cloud setup as I said earlier?
- icedchai 1y agoWhy write one when there are tools like “restic”?
- mikehotel 1y agoDecryption is not usually an issue if you encrypt locally. Tools like Kopia, Borg and Restic handle this and also include deduplication and other advanced features. Really no excuse for large orgs or even small businesses and somewhat tech literate public.
- AshamedCaptain 1y agoIs encryption, almost any form, really reliable protection for a countries' government entire data? I mean, this is _the_ ultimate playground for "state level actors" -- if someday there's a hole and it turns out it takes only 20 years to decrypt the data with a country-sized supercomputer, you can bet _this_ is what multiple alien countries will try to decrypt first.
- lucb1e 1y agoYou're assuming that this needs to protect... > ... a countries' government entire data? But the bulk of the data is "boring": important to individuals, but not state security ("sorry Jiyeong, the computer doesn't know if you are a government employee. Apologies if you have rent to make this month!") There likely exists data where the risk calculation ends up differently, so that you wouldn't store it in this system. For example, for nuke launch codes, they might rather lose than loose them. Better to risk having to reset and re-arm them than to have them hijacked > Is encryption, [in?] any form, really reliable protection There's always residual risk. E.g.: can you guarantee that every set of guards that you have watching national datacenters is immune from being bribed? Copying data around on your own territory thus also carries risks, but you cannot get around it if you want backups for (parts of) the data People in this thread are discussing specific cryptographic primitives that they think are trustworthy, which I think goes a bit deeper than makes sense here. Readily evident is that there are ciphers trusted by different governments around the world for their communication and storage, and that you can layer them such that all need to be broken before arriving at the plain, original data. There is also evidence in the Snowden archives that (iirc) e.g. PGP could not be broken by the NSA at the time. Several ciphers held up for the last 25+ years and are not expected to be broken by quantum computers either. All of these sources can be drawn upon to arrive at a solid choice for an encryption scheme
- makeitdouble 1y agoA foreign gov getting all your security researchers and staff's personal info with their family and tax and medical records doesn't sound great. That's just from the top of my head. Exploiting such a trove of data doesn't sound complicated.
- kazinator 1y agoYou and I can encrypt our data before saving it into the cloud, because we have nothing of value or interest to someone with the resources of a state. Sometimes sensitive data at the government level has a pretty long shelf life; you may want it to remain secret in 30, 50, 70 years.
- waterTanuki 1y agoI don't see how this is any different than countries putting significant portions of their gold & currency reserves in the NY Federal Reserve Bank. If for some reason the U.S. just decided to declare "Your monies are all mine now" the effects would be equally if not more devastating than a data breach.
- don_esteban 1y agoExactly that happened to Russia, Iran, Venezuela
- kazinator 1y agoNot North Korea though; they just have hundreds of thousands of dollars of unpaid parking tickets invested in the USA, which is a negative. https://www.nbcnewyork.com/news/local/north-korea-parking-tickets-united-nations-new-york-diplomat-vehicle-investigation/283490/ https://www.nbcnewyork.com/news/local/north-korea-parking-ti... [2017]
- tsimionescu 1y agoThe difference is that there are sometimes options to recover the money, and at least other countries will see and know that this happened, and may take some action. A data breach, however, is completely secret - both from you and from others. Another country (not even necessarily the one that is physically hosting your data) may have access to your data, and neither you nor anyone else would necessarily know.
- Den_VR 1y agoOn the Microsoft side CVE-2025–55241 is still pretty recent. https://news.ycombinator.com/item?id=45282497 https://news.ycombinator.com/item?id=45282497
- politelemon 1y agoS3 features have saved our bacon a number of times. Perhaps your experience and usage is different. They are worth trusting with business critical data as long as you're following their guidance. GCP though have not proven it, their data loss news is still fresh in my mind.
- hosh 1y agoWere you talking about this incidence? https://arstechnica.com/gadgets/2024/05/google-cloud-accidentally-nukes-customer-account-causes-two-weeks-of-downtime/ https://arstechnica.com/gadgets/2024/05/google-cloud-acciden... I am currently evaluating between GCP and AWS right now.
- Imustaskforhelp 1y agoI read the article and it seems that, that thing happened because their account got deleted and here is something from the article you linked Google Cloud is supposed to have safeguards that don't allow account deletion, but none of them worked apparently, and the only option was a restore from a separate cloud provider (shoutout to the hero at UniSuper who chose a multi-cloud solution). If you are working with really important software, please follow the 3-2-1 EVEN with cloud providers I suppose if you genuinely want ABSOLUTE guarantee I suppose, but it depends on how important the data is I suppose for the prices. I have thought about using some cheap like backblaze and wasabi and others for the 3-2-1 for backups I suppose I am not sure but I do think that this incident was definitely a bit interesting to read into and I will read more about it, I do remember it from kevin fang's video but this article is seriously good and I will read it later, bookmarked.
- eCa 1y agoAgree completely that it's absolute wild to run such a system without backups. But at this point no government should keep critical data on foreign cloud storage.
- stogot 1y agoWhy not? If the region is in country, encrypted, and with proven security attestations validated by third parties, a backup to a cloud storage would be incredibly wise. Otherwise we might end up reading an article about a fire burning down a single data center
- g-b-r 1y agoAnd which organization has every file, from each of their applications using the cloud, encrypted *before* it is sent to the cloud?
- crazygringo 1y agoExactly. Like, don't store it in the cloud of an enemy country of course. But if it's encrypted and you're keeping a live backup in a second country with a second company, ideally with a different geopolitical alignment, I don't see the problem.
- OvbiousError 1y agoEnemy country in the current geopolitical climate is an interesting take. Doesn't sound like a great idea to me tbh.
- fabian2k 1y agoUsing the cloud would have been the easiest way to achieve the necessary redundancy, but by far not the only one. This is just a flawed concept from the start, with no real redundancy.
- DarkmSparks 1y agoBut not security. And for governmental data security is a far more important consideration. not losing data and keeping untrusted parties out of your data is a hard problem, that "cloud" aka "stored somewhere that is accessible by agents of a foreign nation" does not solve.
- freehorse 1y agoAs OP says, cloud is not the only solution, just the easiest. They should probably have had a second backup in a different building. It would probably require a bit more involvement, but def doable.
- DrewADesign 1y agoIt's the government of South Korea, which has a nearly 2 trillion dollar GDP. Surely they could have built a few more data centers connected with their own fiber if they were that paranoid about it.
- miken123 1y agoBecause these companies never lose data, like during some lightning strikes, oh wait: https://www.bbc.com/news/technology-33989384 https://www.bbc.com/news/technology-33989384 As a government you should not be putting your stuff in an environment under control of some other nation, period. That is a completely different issue and does not really relate to making backups.
- whatevaa 1y agoYeah, I heard that consumer clouds are only locally redundant and there aren't even backups. So big DC damage could result in data loss.
- lima 1y agoWhat do you mean by "consumer clouds"?
- alwa 1y agoI mean… at the risk of misinterpreting sarcasm— Except for the backup strategy said consumers apply to their data themselves, right? If I use a service called “it is stored in a datacenter in Virginia” then I will not be surprised when the meteor that hits Virginia destroys my data. For that reason I might also store copies of important things using the “it is stored in a datacenter in Oregon” service or something.
- 1y ago
- firesteelrain 1y agoI know there is legit hate for VMWare/Broadcom but there is a legit case to be made for VCF with an equivalent DR setup where you have replication enabled by Superna and Dell PowerProtect Data Domain protecting both local and remote with Thales Luna K160 KMIP for the data at rest encryption for the vSAN. To add, use F710s, H710s and then add ObjectScale storage for your Kubernetes workloads. This setup repatriates your data and gives you a Cloud like experience. Pair it with like EKS-A and you have a really good on premises Private Cloud that is resilient.
- threeducks 1y agoThis reads very similar to the Turbo Encabulator video.
- pico303 1y agoWhat a lame excuse. “The G-Drive’s structure did not allow for backups” is a blatant lie. It’s code for, “I don’t value other employees’ time and efforts enough to figure out a reliable backup system; I have better things to do.” Whoever made this excuse should be demoted to a journeyman ops engineer. Firing would be too good for them.
- CoastalCoder 1y agoYou could be right, but it could also be a bad summary or bad translation. We shouldn't rush to judgement.
- MBCook 1y agoIt could be accurate. Let’s say, for whatever reason, it is. Ok. Then it wasn’t a workable design. The idea of “backup sites” has existed forever. The fact you use the word “cloud” to describe your personal collection of servers doesn’t suddenly mean you don’t need backups in a separate physical site. If the government mandates its use, it should have a hot site at a minimum. Even without that a physical backup in a separate physical location in case of fire/attack/tsunami/large band of hungry squirrels is a total must-have. However it was decided that not having that was OK, that decision was negligence.
- sph 1y agoSilly to think this is the fault of ops engineers. More likely, the project manager or C-suite didn't have time nor budget to allocate on disaster recovery. The project shipped, it's done, they've already moved us onto the next task, no one wants to pay for maintenance anyway. This has been my experience in 99% of the companies I have worked for in my career, while the engineers that built the bloody thing groan and are well-aware of all the failure modes of the system they've built. No one cares, until it breaks, and hopefully they get the chance to say "I **** told you this was inadequate"
- littlestymaar 1y ago> The government official who insisted that commercial AWS/GCP/Azure couldn't possibly be trusted with keeping the information They were still right though: it's absolutely clear without an ounce of doubt that whatever you put on an US cloud is being accessible by the US government, who can also decide to sanction you and deprive you from your ability to access the data yourself. Not having backups is entirely retarded, but also completely orthogonal.
- otterley 1y agoThe U.S. Government can’t decrypt data for which it does not possess the key (assuming the encryption used is good).
- dboreham 1y agoIn theory. I'm very much happier to have my encrypted data also not be available to adversaries.
- littlestymaar 1y agoWell first of all neither you and I knows the decryption capabilities of the NSA, all we know is that they have hired more cryptologists than the rest of the world combined. Also, it's much easier for an intelligence service to get the hand on a 1kB encryption key than on a PB of data: the former is much easier to exfiltrate without being noticed. And then I don't know why you bring encryption here: pretty much none of the use-case for using a cloud allow for fully encrypted data. (The only one that does is storing encrypted backups on the cloud, but the issue here is that the operator didn't do backups in the first place…)
- otterley 1y ago1. More evidence suggests that NSA does not know how to decrypt state-of-the-art ciphers than suggests they do. If they did know, it's far less likely we'd have nation states trying to force Apple and others to provide backdoors for decryption of suspects' personal devices. (Also, as a general rule, I don't put too much stock in the notion that governments are far more competent than the private sector. They're made of people, they pay less than the private sector, and in general, if a government can barely sneeze without falling over, it's unlikely they can beat our best cryptologists at their own game.) 2. The operative assumption in my statement is that the government does not possess the key. If they do possess it, all bets are off. 3. This thread is about a hypothetical situation in which the Korean government did store backups with a U.S.-based cloud provider, and whether encryption of such backups would provide adequate protection against unwanted intrusion into the data held within.
- alwa 1y agoNot sure “sane backup strategy” and “park your whole government in a private company under American jurisdiction” are mutually exclusive. I feel like I can think of a bunch of things that a nation would be sad to lose, but would be even sadder to have adversaries rifling through at will. Or, for that matter, extort favors under threat of cutting off your access. At least in this case you can track down said officials in their foxholes and give them a good talking-to. Good luck holding AWS/GCP/Azure accountable…
- atoav 1y agoWell it is just malpractise. Even when I was an first semester art student I knew about the concept of off-site backups.
- Nux 1y agoHe may or may not have been right, but it's besides the point. The 3-2-1 backup rule is basic.
- StopDisinfo910 1y agoThe issue here is not refusing to use a foreign third party. That makes sense. The issue is mandating the use of remote storage and not backing it up. That’s insane. It’s like the most basic amount of preparation you do. It’s recommended to even the smallest of companies specifically because a fire is a risk. That’s gross mismanagement.
- VirusNewbie 1y ago[flagged]
- lukevp 1y agoHow’s that? Using encryption, which is known to have backdoors and is vulnerable to nation state cracking?
- vbezhenar 1y agoIt is much more likely and cheaper, that US marines will desant and capture your backup facility, than someone would break AES-128.
- senko 1y agoSending troops would be an act of war, and definitely not cheap. Stealing some encryption keys, just another Wednesday.
- deleted 1y ago[deleted]
- VirusNewbie 1y agoYou mean like blowing up an oil pipeline? Accidents happen all the time. It is quite a lot cheaper to have an 'accident' happen to a data center than to break AES-256.
- lofaszvanitt 1y ago
- speedgoose 1y agoYeah let’s fax all government data to the Trump administration.
- deleted 1y ago[deleted]
- PunchyHamster 1y agocloud will also not back up your stuff if you configure it wrong so not sure how's that related
- juancb 1y agoThe simple solution here would have been something like a bunch of netapps with snapmirrors to a secondary backup site. Or ZFS or DRBD or whatever homegrown or equivalent non-proprietart alternative is available these days and you prefer.
- rr808 1y agoThere is some data privacy requirement in SK where application servers and data have to remain in the country. I worked for a big global bank and we had 4 main instances of our application: Americas, EMEA, Asia and South Korea.
- chatmasta 1y agoIf only there were a second data center in South Korea where they could backup their data…
- bradly 1y agoWhen I worked on Apple Maps infra South Korea required all servers be in South Korea.
- srj 1y agoIt was the same at google. If I'm remembering right we couldn't export any vector type data (raster only) and the tiles themselves had to be served out of South Korea.
- kumarvvr 1y ago> The government official who insisted that commercial AWS/GCP/Azure couldn't possibly be trusted with keeping the information will be keeping their head low for a few days then... They absolutely cannot be trusted, especially sensitive govt. data. Can you imagine the US state department getting their hands on compromising data on Korean politicians? Its like handing over the govt. to US interests wholesale. That they did not choose to keep the backup, and then another, at different physical locations is a valuable lesson, and must lead to even better design the next time. But the solution is not to keep it in US hands.
- waterTanuki 1y agoI understand data sovereignty in the case where a foreign entity might cut off access to your data, but this paranoia that storing info under your bed is the safest bet is straight up false. We have post-quantum encryption widely available already. If your fear is that a foreign entity will access your data, you're technologically illiterate. Obviously no person in a lawmaking position will ever have the patience or foresight to learn about this, but the fact they won't even try is all the more infuriating.
- franga2000 1y agoEncryption only makes sense if "the cloud" is just a data storage bucket to you. If you run applications in the cloud, you can't have all the data encrypted, especially not all the time. There are some technologies that make this possible, but none are mature enough to run even a small business, let alone a country on. It sounds technologically illiterate to you because when people say "we can't safely use a foreign cloud" you think they're saying "to store data" and everyone else is thinking at the very least "to store and process data". Sure, they could have used a cloud provider for encrypted backups, but if they knew how to do proper backups, they wouldn't be in this mess to begin with.
- TulliusCicero 1y ago> G-Drive’s structure did not allow for external backups Ha! "Did not allow" my ass. Let me translate: > We didn't feel like backing anything up or insisting on that functionality.
- p0w3n3d 1y agoDays? That's optimistic. It depends on what govt cloud contained. For example imagine all the car registrations. Or all the payments for the pension fund
- j45 1y agoThey put everything only in one datacenter. A datacenter located elsewhere should have been setup to mirror. This has nothing to do with commercial clouds. Commercial clouds are just datacenters. They could pick one commercial cloud data center and not do much more to mirror or backup in different regions. I understand some of the services have inherent backups.
- NetMageSCW 1y agoMirroring is not backup.
- raxxorraxor 1y agoPretty sensible to not host it on these commercial services. What is not so sensible is to not make backups.
- didntknowyou 1y agoyour first criticism was they should have handed their data sovereignty over to another country? there are many failure points here, not paying Amazon/Google/Microsoft is hardly the main point.
- INTPenis 1y agoDude, the issues go wayyy beyond opting for selfhosting rather than US clouds. We use selfhosting, but we also test our fire suppression system every year, we have two different DCs, and we use S3 backups out of town. Whoever runs that IT department needs to be run out of the country.
- ubermonkey 1y agoI was once advised to measure your backup security in zip codes and time zones. You have a backup copy of your file, in the same folder? That helps for some "oops" moments, but nothing else. You have a whole backup DRIVE on your desktop? That's better. Physical failure of the primary device is no longer a danger. But your house could burn down. You have an alternate backup stored at a trusted friend's house across the street? Better! But what if a major natural disaster happens? True life, 30+ years ago when I worked for TeleCheck, data was their lifeblood. Every week a systems operator went to Denver, the alternate site, with a briefcase full of backup tapes. TeleCheck was based in Houston, so a major hurricane could've been a major problem.
- CarlitosHighway 1y agoI mean he's still right about AWS etc. with the current US Administration and probably all that will follow - but that doesn't excuse not keeping backups.
- stonemetal12 1y agoIf you (as the SK government) were going to do a deal with " AWS/GCP/Azure" to run systems for the government, wouldn't you do something like the Jones Act? The datacenters must be within the country and staffed by citizens, etc.
- dh2022 1y agoMicrosoft exec testified that US Govt can get access to the data Azure stores in other countries. I thought this was a wild allegation but apparently is true [0]. [0]https://www.theregister.com/2025/07/25/microsoft_admits_it_cannot_guarantee/ https://www.theregister.com/2025/07/25/microsoft_admits_it_c...
- lumost 1y agoUsually these mandates are made by someone who evaluates “risks.” Third party risks are evaluated under the assumption that everything will be done sensibly in the 1p scenario, to boot, the 1p option will be cheaper as disk drives etc are only a fraction of total cost. Reality hits later when budget cuts/constrained salaries prevent the maintenance of a competent team. Or the proposed backup system is deemed as excessively risk averse and the money can’t be spared.
- harikb 1y ago"Not my fault.. I asked them to save everything in G-Drive (Google Drive)"
- delfinom 1y ago>The government official who insisted that commercial AWS/GCP/Azure couldn't possibly be trusted with keeping the information will be keeping their head low for a few days then... They can't. The trump admin sanctioning the international criminal court and Microsoft blocking them from all services as a result are proof of why.