20 ms·
Fire destroys S. Korean government's cloud storage system, no backups available
https://www.chosun.com/english/national-en/2025/10/02/FPWGFSXMLNCFPIEGWKZF3BOQ3M/ https://www.chosun.com/english/national-en/2025/10/02/FPWGFS...
- EverydayBalloon 1y ago[dead]
- benoau 1y ago> However, due to the system’s large-capacity, low-performance storage structure, no external backups were maintained — meaning all data has been permanently lost. Yikes. You'd think they would at least have one redundant copy of it all. > erasing work files saved individually by some 750,000 civil servants > 30 gigabytes of storage per person That's 22,500 terabytes, about 50 Backblaze storage pods. Or even just mirrored locally.
- yongjik 1y agoIt's even worse. According to other articles [1], the total data of "G drive" was 858 TB. It's almost farcical to calculate, but AWS S3 has pricing of about $0.023/GB/month, which means the South Korean government could have reliable multi-storage backup of the whole data at about $20k/month. Or about $900/month if they opted for "Glacier deep archive" tier ($0.00099/GB/month). They did have backup of the data ... in the same server room that burned down [2]. [1] https://www.hankyung.com/article/2025100115651 https://www.hankyung.com/article/2025100115651 [2] https://www.hani.co.kr/arti/area/area_general/1221873.html https://www.hani.co.kr/arti/area/area_general/1221873.html (both in Korean)
- paleotrope 1y agoThat's unfortunate.
- deleted 1y ago[deleted]
- Forgeties79 1y agoCouldn’t even be bothered to do a basic 3-2-1! Wow
- sneak 1y agoDid you expect government IT in a hierarchical respect-your-superiors-even-when-wrong society to be competent?
- Forgeties79 1y agoI mean...I feel you but holy hell dude. Nothing? Boggles the mind. Edit: my bad backups in the room is something, somehow just forgot about that part
- username332211 1y agoSouth Korea isn't some sort of backwards nation and I'm sure it's chaebols share the same culture. Having had unfortunate encounters with government IT in other countries I can bet that the root cause wasn't the national culture. It was the internal culture of "I want to do the same exact same thing I've always done until the day I retire." Absent outside pressure, civil services across the word tend advance scientifically - one funeral (or retirement) at a time.
- deleted 1y ago[deleted]
- rvba 1y agoHow does this even make sense business wise for AWS? Is their cost per unit so low?
- Ekaros 1y agoWhen you start to do math, hard drive are cheap when you go for capacity and not performance. 0.00099*1000 is 0.99. So about 12$ a year. Now extrapolate something like 5 year period or 10 year period. And you get to 60 to 120$ for TB. Even at 3 to 5x redundancy those numbers start to add up.
- vbezhenar 1y agoS3 does not spend 3x drives to provide redundancy. Probably 20% more drives or something like that. They split data to chunks and use erasure coding to store them in multiple drives with little overhead.
- hapanin 1y agowait, can you elaborate on how this works?
- vbezhenar 1y agoYou have 100 bytes file. You split it into 10 chunks (data shards) and add 11-th chunk (parity shard) as XOR of all 10 chunks. Now you store every chunk on separate drive. So you have 100 bytes and you spent 110 bytes to store them all. Now you can survive one drive death, because you can recompute any missing chunk as XOR of all chunks. That's very primitive explanation, but should be easy to understand. In reality S3 uses different algorithm (probably Reed-Solomon codes) and some undisclosed number of shards (probably different for different storage classes). Some say that they use 5 of 9 (so 5 data shards + 4 parity shards which makes for 80% overhead), but I don't think it's official information.
- gundmc 1y agoAFAIK geo-replication between regions _does_ replicate the entire dataset. It sounds like you're describing RAID configurations, which are common ways to provide redundancy and increased performance within a given disk array. They definitely do that too, but within a zone
- mastax 1y agoI made an 840TB storage server last month for $15,000.
- ycombinatrix 1y ago840TB before or after configuring RAID?
- mastax 1y ago840TB raw unformatted.
- maxlin 1y agoI have almost 10% of that in my closet RAID5'd with large part of it backing up constantly to Backblaze for 10$/month, running on 10 year old hardware, with basically only the hard drives having any value ... Used a case made of cardboard till I wanted to improve the cooling, and got a used Fractal Design case for 20€. _Only_ the kind of combination of incompetence and bad politics here can lead to the kind of % of how much data has been lost here, given the policy was to only save stuff on that "G-drive" and avoid local copies. The "G-drive" they intentionally did not back up because they couldn't figure out a solution to at least store a backup across the street ...
- rasz 1y ago>AWS S3 has pricing of about $0.023/GB/month, which means ... about $20k/month or outright buying hardware capable of storing 850TB for the same $20K one time payment. Gives you some perspective on how overpriced AWS is.
- swarnie 1y agoWhere are you getting 850TB of enterprise storage for $20k? I had 500TB of object storage priced last year and it came out closer to $300k
- kachapopopow 1y ago136tb for $3k (used gen 2 epyc hardware and refurb <1 hour 16tb hdd's) they're zero risk after firmware validation and 1 full drive read and write.
- HHad3 1y agoThat's including the enterprise premium for software, hardware support, and licenses. Building this in-house using open source software (e.g. Ceph) on OEM hardware will be cheaper by an order of magnitude. You of course need people to maintain it -- the $300k turnkey solution might be the better option depending on current staff.
- mort96 1y agoPriced out by whom? What kind of object storage? Were you looking at the price of drives, or the price of a complete solution delivered by some company?
- jeroenhd 1y agoAWS? Linus Tech Tips has run multiple petabyte servers in their server closet just for sponsor money and for the cool of it. No need to outsource your national infrastructure to foreign governments, a moderate (in government terms) investment in a few racks across the country could've replicated everything for maybe half a year's worth of Amazon subscription fees.
- TheRoque 1y agoExactly, everyone here on hackernews is talking about Azure/AWS/GCP as if it was the only correct way to store data. Americans are too self centered, it's quite crazy.
- mrbadguy 1y agoYeah the comments here are slightly surreal; the issue was that they didn’t have an off-site backup at all, not that it wasn’t on AWS or whatever.
- axus 1y agoBut then they will depend on the security people at both sides to agree on the WAN configuration. Easier to let everything burn in a fire and rebuild from scratch.
- deleted 1y ago[deleted]
- deleted 1y ago[deleted]
- baobabKoodaa 1y agoYou're assuming average worker utilized the full 30G of storage. More likely average was at like 0.3G.
- jeroenhd 1y agoOn the other hand: backups should also include a version history of some kind, or you'd be vulnerable to ransomware.
- PeterStuer 1y agoI'm sure they had dozens of process heavy cybersecurity committees producing hundreds if not thousands of powerpoints and word documents outlining procedures and best practices over the last decade. There is this weird divide between the certified class of non-technical consultants and actual overworked and pushed to corner cut techs.
- zaphar 1y agoIronically many of those documents for procedures probably lived on that drive...
- ksec 1y agoI dont know why but cant stop laughing. And the great thing is that they will get paid again to write the same thing.
- comprev 1y agoYou jest, but I once had a client who's IaC provisioning code was - you guessed it - stored on the very infrastructure which got destroyed.
- __turbobrew__ 1y agoIf you are one of the big boys (FAANG and other large companies who run physical infra) you will have this problem as well. The infra systems run and replace themselves and if something fundamental breaks (for example, your deployment system requires DNS, but your DNS servers are broken, but you cannot deploy to fix them as the deploy service requires DNS). From what I have seen a lot of time the playbooks to fix these issues are just rawdogging files using rsync manually. Ideally you deploy your infrastructure in cells where rollouts proceed cell by cell so you can catch issues sooner and also implement failover to bootstrap broken cells (in my DNS example, client could talk to DNS servers in the closest non-broken cell using BGP based routing). It is hard to test, and there are some global services (like that big Google outage a few months ago was due to the global auth service being down).
- 1y ago
- Titan2189 1y agoSurely there must be something that's missing in translation? This feels like it simply can't be right.
- mrbluecoat 1y agoI agree. No automated fire suppression system for critical infrastructure with no backup?
- fredoralive 1y agoThat may not be a perfect answer. One issue with fire suppression systems and spinning rust drives is that the pressure change etc. from the system can also ‘suppress’ the glass platters in drives as well.
- privatelypublic 1y agoI'd be interested in if you can even use dry fire suppression on the 5th floor of a building.
- magicalhippo 1y agoReminds me of the classic video[1] showing how shouting at the harddrives make them go slower. [1]: https://www.youtube.com/watch?v=tDacjrSCeq4 https://www.youtube.com/watch?v=tDacjrSCeq4
- perlgeek 1y agoThat's why the top-security DCs that my employer operates have large quantities of Nitrogen stored, and use that slightly lower the O2 saturation of the air in the case of fire. Yes, it's fucking expensive, that's one of the reason you pay more for a VM (or colocation) than at Hetzner or OVH. But I'm also pretty confident that single fire wouldn't destroy all hard drives in that IT space.
- rini17 1y agoBattery fire is impossible to suppress.
- BrandoElFollito 1y ago> all documents be stored exclusively on G-Drive Does G-Drive mean Google Drive, or "the drive you see as G:"? If this is Google Drive, what they had locally were just pointers (for native Google Drive docs), or synchronized documents. If this means the letter a network disk storage system was mapped to, this is a weird way of presenting the problem (I am typing on the black keyboard and the wooden table, so that you know)
- bryanhogan 1y agoSaw a few days ago that the application site for the GKS, the most important scholarship for international students in Korea, went offline for multiple days, surprising to hear that they really lost all of the data though. Great opportunity to build a better website now? But yeah it's a big problem in Korea right now, lots of important information just vanished, many are talking about it.
- Zacharias030 1y agoMust have been a program without much trickle down into gov tech
- m3047 1y agoMindblowing. Took a walk. All I can say is that if business continues "as usual" and the economy and public services continue largely unaffected then either there were local copies of critical documents, or you can fire a lot of those workers; either one of those ways the "stress test" was a success.
- layer8 1y ago“Final reports and official records submitted to the government are also stored in OnNara, so this is not a total loss”.
- dghlsakjg 1y agoHow do you come to the conclusion that because things work without certain documents that you can start laying off workers?
- RaptorJ 1y agoSurely having human-resource backups will also help with disaster recovery
- MiddleEndian 1y ago>or you can fire a lot of those workers Sometimes things can seem to run smoothly for years when neglected... until they suddenly no longer run smoothly!
- npteljes 1y agoLong term damage, and risk are two things that don't show up with a test like this. Also, often why things go forward is just momentum, built from the past.
- danparsonson 1y agoYeah you can do the same with your car too - just gradually remove parts and see what's really necessary. Seatbelts, horn, rear doors? Gone. Think of the efficiency!
- rotis 1y agoThe fire started on 26th September and news about it reached HN only now. I think this is telling how disruptive for South Korea daily life this accident really was.
- aio2 1y agoFunny, because the same thing happened in Nepal a few weeks ago. Protestors/rioters burned some government buildings, along with the tech infrastructure within them, so now almost all electronic data is gone.
- cyanydeez 1y ago[flagged]
- dottjt 1y agoWould this have been any different if these documents were stored non-electronically though? I understand that the whole point of electronic data is that it can be backed up, but if the alternative were simply an analog system then it would have fared no better.
- seunosewa 1y agoIt would have been better if storage was distributed.
- Muromec 1y agoPaper records are usually distributed both by agency and by locality.
- dikei 1y agoFor paper documents, you'd make at least a few copies for storage at the source, and then every receiver will get his/her own notarized copies. Electronically, everyone just receives a link to read the document.
- rvba 1y agoHappened in Bladerunner too
- senordevnyc 1y agoAnd Fight Club
- 727564797069706 1y agoMeanwhile, Estonia has a "data embassy" in Luxembourg: https://e-estonia.com/solutions/e-governance/data-embassy/ https://e-estonia.com/solutions/e-governance/data-embassy/ TL;DR: Estonia operates a Tier 4 (highest security) data center in Luxembourg with diplomatic immunity. Can actively run critical government services in real-time, not just backups.
- lostmsu 1y agoThis comment is in some way more interesting than the topic of the article.
- _joel 1y agoTotally, backup disasters are a regular occurence (maybe not to the degree of negligence) but the Estonia DR is wild.
- lucb1e 1y agoDefinitely. Especially when considering that there were 95 other systems in this datacentre which do have backups and > The actual number of users is about 17% of all central government officials Far from all, and they're not sure what's recoverable yet ("“It’s difficult to determine exactly what data has been lost.”") Which is not to say that it's not big news ("the damage to small business owners who have entered amounts to 12.6 billion Korean won.” The ‘National Happiness Card,’ used for paying childcare fees, etc., is still ‘non-functional.’"), but to put it a bit in perspective and not just "all was lost" as the original submission basically stated Quotes from https://www.chosun.com/english/national-en/2025/10/02/FPWGFSXMLNCFPIEGWKZF3BOQ3M/ https://www.chosun.com/english/national-en/2025/10/02/FPWGFS... as linked by u/layer8 elsewhere in this thread
- hkt 1y agoThat is absolutely delightful. Estonia is just _good_ at this stuff. Admirable.
- lukeqsee 1y agoThis is because everything is in digital form. Essentially all government systems are digital-first, and for the citizen, often digital-only. If the data is lost, there may be no paper records to restore everything from land registry, business registry (operating agreements, ownership records), etc. Without an out-of-country backup, a reversion to previous statuses means the country is lost (Estonia has been occupied a lot). With it, much of the government can continue to function, as an expat government until freedom and independence is restored.
- layer8 1y agoSome more details in this article: https://www.chosun.com/english/national-en/2025/10/02/FPWGFSXMLNCFPIEGWKZF3BOQ3M/ https://www.chosun.com/english/national-en/2025/10/02/FPWGFS...
- dang 1y agoThanks! we've added that link to the toptext as well
- lucb1e 1y ago> The stored data amounts to 858TB (terabytes), equivalent to 449.5 billion A4 sheets. This attempt at putting it in perspective makes me wonder what would put it in perspective. "100M sets of harry potter novels" would be one step in the right direction, but nobody can imagine 100M of anything either. Something like "a million movies" wouldn't work because they are very different from text media in terms of how much information is in one, even if the bulk of the data is likely media. It's an interesting problem even if this article's attempt is so bad it's almost funny Good article otherwise though, indeed a lot more detail than the OP. It should probably replace the submission. Edit: dang was 1 minute faster than me :)
- sixothree 1y ago"equivalent to 50 hard drives" ?
- lucb1e 1y agoI don't think many people have 16TB of storage on the hard drives they're familiar with but I take your point nonetheless! Simple solution. I was apparently too focused on the information factor and didn't think of just saying how many of some storage medium it is equivalent to, which makes a lot of sense indeed
- mort96 1y agoHow about "equivalent to 858 1TB hard drives"?
- jopsen 1y ago> The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for external backups. This is why I don't really want to run my own cloud :) Actually testing the backups is boring. That said, ones the flames are out, they might actually be able to recover some of it.
- whartung 1y agoTesting backups is boring. If you want exciting, test restores!
- Imustaskforhelp 1y agoHm, care to elaborate. I kinda liked this idea even though I know that it shouldn't make much sense but still lol, would this have any benefits over testing backups other than the excitement lol
- procaryote 1y agoThe joke is that if you don't test backups you'll end up seeing if they work once you're restoring after a disaster, which is exciting
- fer 1y agoI'm stealing this!
- MangoCoffee 1y agowhat's the point of a storage system with no back up?
- WJW 1y agoIt works fine as long as it doesn't break, and it's cheaper to buy than an equivalently sized system that does have back ups.
- lucb1e 1y agoIsn't that self-evident? Do you have two microwaves from different batches, regularly tested, solely for the eventuality that one breaks? Systems work fine until some (unlikely) risk manifests... Idk if this sounds like I'm against backups, I'm not, I'm just surprised by the question
- MangoCoffee 1y agoIt's hard to believe this happened. South Korea has tech giants like Samsung, and yet this is how the government runs? Is the US government any better?
- userbinator 1y agoThe US government still relies heavily on physical records.
- nebula8804 1y agoDidn't Elon shut that down? [0]: https://www.cnbc.com/2025/02/13/company-ripped-by-elon-musk-that-stores-government-records-in-a-mine-may-actually-get-a-boost-from-doge.html https://www.cnbc.com/2025/02/13/company-ripped-by-elon-musk-...
- AshamedCaptain 1y agoWhy is there a "still" in there?
- ashirviskas 1y agoSouth Korean IT seemed to be stuck in 2007 just not too long ago, would be surprised if it has changed much in the last few years. Do the websites still require you to use internet explorer?
- r_lee 1y agoSoftware and information technology in Korea just sucks. buttons are jpegs/gifs, everything is on Java EE and on vulnerable old webservers etc... A lot of government stuff supports only Internet Explorer even though it's long dead
- creakingstairs 1y agoRemember Log4j vulnerability? A lot of the Korea governmental sites weren't affected because the Java version was too old :) Don't even get me started on ActiveX.
- aorloff 1y agoTheoretically, they still have the primary copies (on each individual person's "cloud-enabled" device).
- cthalupa 1y ago> The Ministry of the Interior and Safety also issued guidelines to each ministry stating, “All work materials should not be stored on office PCs but should be stored on the G-Drive.” They very well might have only been saving to this storage system. It was probably mapped as a drive or shared folder on the PC.
- crazygringo 1y agoDo they? It's not clear if this was two-way sync or access on-demand. Like, I use Google Drive for Desktop but it only downloads the files I access. If I don't touch a file for a few days it's removed from my local cache.
- jn78 1y agohttps://phrack.org/issues/72/7_md#article https://phrack.org/issues/72/7_md#article
- msbhvn 1y agoWoah, read the timeline at the top of this. The fire happened the very day the government ordered onsite inspection was supposed to start due to Chinese/NK hacking.
- yieldcrv 1y agoSo, someone figured out how to do backups
- southernplaces7 1y agoThey certainly will after this.
- AnimalMuppet 1y agoYeah, this whole thing smells. Who has the incentive to do this, though? China/North Korea? Or someone in South Korea trying to cover up how bad they messed up? Does adding this additional mess on top mean they looked like they messed up less? (And for that to be true, how horrifically bad does the hack have to be?)
- mattmaroon 1y agoIt might be different “they”s. Putting on my tinfoil hat, whoever was going to be in hot water over the hack burns it down and now the blame shifts from them to whoever manages G-drive and don’t have a backup plan. Not saying I believe this (or even know enough to have an opinion), but it’s always important to not anthropomorphize a large organization. The government isn’t one person (even in totalitarian societies) but an organization that contains large numbers of people who may all have their own motivations.
- dang 1y ago[stub for offtopicness]
- mouse_ 1y agoWe will learn nothing
- pr337h4m 1y agoNow imagine they had a CBDC.
- glitchc 1y agoI thought most liberal governments gave up on those.
- throwaway984393 1y ago[dead]
- blueflow 1y ago[flagged]
- oldpersonintx2 1y ago[dead]
- dvh 1y agoTechnically the data is still in the cloud
- ivape 1y agoI mean ... was making backups on the backlog at least? Can they at least point to the work item that was going to get done soonish?
- redditor98654 1y agoMay be a “fast follow”? Right after launch of the “MVP”?
- odie5533 1y agoIt got pushed a couple sprints and we've got it on the plan for next quarter as long as no new features come in before then.
- deleted 1y ago[deleted]
- FinnKuhn 1y agoIf it wasn't it most certainly is now
- system2 1y agoI wonder how many IT professionals were begging some incompetent upper management official to do this the right way, but were ignored daily. You'd think there would be concrete policies to prevent these things...
- zulban 1y agoIf I worked there I'd have had a hard time believing there were really no backups. Governments can be very nebulous.
- kristianc 1y agoThe government official who insisted that commercial AWS/GCP/Azure couldn't possibly be trusted with keeping the information will be keeping their head low for a few days then... "The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for external backups." This is absolutely wild.
- zwnow 1y agoRightfully did not trust these companies. Sure what happened is a disaster for them, but you cant simply trust Amazon & Microsoft.
- oceansky 1y agoFor sure the only error here is zero redundancy.
- kingnothing 1y agoWhy not? You can easily encrypt your data before sending it for storage on on S3, for example.
- zhouzhao 1y agoYou can encrypt them at rest, but data that lies encrypted and is never touched, is useless data. You need to decrypt them as well. Also, plenty of incompetent devops around, and writing a decryption toolchain can be difficult.
- kspacewalk2 1y agoAm I missing something? If you ever need to use this data, obviously you transfer it back to your premises and then decrypt it. Whether it's stored at Amazon or North Korean Government Cloud makes no difference whatsoever if you encrypt before and decrypt after transfer.
- redwood 1y agoS. Korea has the most backward infosec requirements. It's wild
- Frost1x 1y agoHaving just visited South Korea last year, one thing that sort of caught me off guard was the lack of Google Maps or other major direction system. I wasn’t aware but turns out anything considered “detailed mapping” infrastructure has to be ran stored and on South Korean soil, probably lots of other requirements. So you’re stuck with some shotty local mapping systems that are just bad. There may be a point in time it made sense but high resolution detailed satellite imagery is plenty accessible and someone could put a road and basically planning structure atop it, especially a foreign nation wishing to invade or whatever they’re protecting against. Some argument may be made that it would be a heavy lift for North Korea but I don’t buy it, incredibly inconvenient for tourists for no obvious reason.
- WhyNotHugo 1y agoSeveral other countries have similar requirements with regards to storing and serving maps locally. If you take a moment to think about it, what's weird is that so many countries have simply resorted to relying on Google Maps for everyday mapping and navigation needs. This has become such a necessity nowadays that relying on a foreign private corporation for it sounds like a liability.
- bmandale 1y agoOSM is competitive with google maps in most places. Even if a person uses google maps, its inaccurate to say they "rely" on it when they could fail over to osm if google maps went down.
- Avamander 1y agoLocal mapping efforts and allowing Google Maps to operate aren't mutually exclusive though. I don't see how it's weird that people can choose which map app they use.
- tedk-42 1y agoA management / risk issue and NOT and engineering one.
- gtirloni 1y agowow https://x.com/koryodynasty/status/1973956091638890499 https://x.com/koryodynasty/status/1973956091638890499 > A senior government official overseeing recovery efforts for South Korea's national network crisis has reportedly died by suicide in Sejong.
- covercash 1y agoIf the US government and corporate executives had even half this level of shame, we'd have nobody left in those positions!
- makeitdouble 1y ago"suicide" in these circumstances is usually something else altogether. Even in cases it is executed by themselves, shame won't be the primary motivation.
- parineum 1y agoIt usually isn't but people do usually imply otherwise.
- spoaceman7777 1y agoYou may want to familiarize yourself more with the culture around this in places like South Korea and Japan.
- makeitdouble 1y agoIt can be posed as shame on the front side. More often than not the suicide covers a whole organization's dirty laundry. You'll have people drunk and driving their cars over cliffs [0], low profile actors ending their life as shit hits the fan [0] etc. Then some on the lower rank might still end their life to spare their family financially (insurance money) or because they're just so done with it all, which I'd put more on depression than anything. Us putting it on shame is IMHO looking at it through rose colored glasses and masking the dirtier reality to make it romantic. [0] https://bunshun.jp/articles/-/76130 https://bunshun.jp/articles/-/76130 [1] https://www.tsukubabank.co.jp/cms/article/a9362e73a19dc0efcfa643278e94ada07b8674a5/ https://www.tsukubabank.co.jp/cms/article/a9362e73a19dc0efcf...
- ookblah 1y agoafter the kakao fire incident and now this i struggle to understand how they got so advanced in other areas. this is like amateur hour level shit.
- pezezin 1y agoIt is the same in Japan. They are really good for hardware and other "physical" engineering disciplines, but they are terrible when it comes to software and general IT stuff. Seriously, I work here as an IT guy and I can't stop wondering how they could become so advance in other areas and stay so backwards in anything software-related except videogames.
- rester324 1y agoYeah. This is my exact experience too wrt japan! The japanese just somehow can't assess and manage neither the scale, nor the complexity, the risk, the effort or the cost of software projects. Working in japan as a software guy feels like working in a country lagging 30-40 years behind :/
- pezezin 1y agoIn my experience, risk assessment is the worst part. Japanese culture is extremely risk averse, and the moment you ask them to do something that they have never done before, they freeze in panic. They need a procedure for everything, and have a really hard time improvising. That's why they are good for industrial processes where they can iterate and improve in small, incremental steps, but terrible for software projects full of uncertainties.
- Theodores 1y agoI was smirking at this until I remembered that I have just one USB stick as my 'backup'. And that was made a long time ago. Recently I have been thinking about whether we actually need governments, nation states and all of the hubris that goes with it such as new media. Technically this means 'anarchism' with everyone running riot and chaos. But, that is just a big fear, however, the more I think through the 'no government' idea, the less ludicrous it sounds. Much can be devolved to local government, and so much else isn't really needed. South Korea's government have kind-of deleted themselves and my suspicion is that, although a bad day for some, life will go on and everything will be just fine. In time some might even be relieved that they don't have this vast data store any more. Regardless, it is an interesting story regarding my thoughts regarding the benefits of no government.
- poncho_romero 1y agoGovernment is whatever has a monopoly on violence in the area you happen to live. Maybe it’s the South Korean government. Maybe it’s a guy down the street. Whatever the case, it’ll be there.
- forinti 1y agoWhat structure could possibly preclude backups? I've never seen anything that couldn't be copied elsewhere. Maybe it was just convenient to have the possibility of losing everything.
- Johnny555 1y agoI think that alluded to that earlier in the article: >However, due to the system’s large-capacity, low-performance storage structure, no external backups were maintained — meaning all data has been permanently lost. I think they decided that their storage was too slow to allow backups? Seems hard to believe that they couldn't manage any backups... other sources said they had around 900TB of storage. An LTO-9 tape drive holds ~20TB uncompressed, so they could have backed up the entire system with 45 tapes. At 300MB/sec with a single drive, it would take them a month to complete a full backup, so seems like even a slow storage system should be able to keep up with that rate. They'd have a backup that's always a month out of date, but that seems better than no backup at all.
- themafia 1y agoToo slow to allow batched backups. Which means you should just make redundant copies at the time of the initial save. Encrypt a copy and send it offsite immediately. If your storage performance is low then you don't need fat pipes to your external provider either. They either built this too quickly or there was too much industry corruption perverting the process and the government bought an off the shelf solution that was inadequate for their actual needs.
- burnt-resistor 1y agoLet's run the numbers: LTO-9 ~$92/tape in bulk. A 4 drive library with 80 drive capacity costs ~$40k* and can sustain about 1 Gbps. It also needs someone to barcode, inventory, and swap tapes once a week and an off-site vaulting provider like Iron Mountain. That's another $100k/year. Also, that tape library will need to be replaced every 4-7 years, so say 6 years. And those tapes wear out over X uses and sometimes go bad too. It might also require buying a server and/or backup/DR software too. Furthermore, a fire-rated data safe is recommended for about 1-2 weeks' worth of backups and spare media. Budget at least $200k/year for off-site tape backups for a minimal operation. (Let me tell you about the pains of self-destructing SSL2020 AIT-2 Sony drives.) If backups for other critical services and this were combined, it would probably be cheaper to scale this kind of service rather reinventing the wheel for just one use-case in one department. That would allow for possibly multiple types of optimizations like network-based backups to nearline storage to then be streamed more directly to tape and using many more tape drives, possibly a tape silo robot(s) and perhaps split into 2-3 backup locations obviating the need for off-site vaulting. Furthermore, it might be simpler, although more expensive, to operate another hot-/warm-site for backups and temporary business continuity restoration using a pile of HDDs and a network connection that's probably faster than that tape library. (Use backups, not replication because replication of errors to other sites is fail.) Or the easiest option is to use one or more cloud vendors for even more $$$ (build vs. buy tradeoff). * Traditionally (~20 years ago), enterprise "retail" prices of gear was sold at around 100% markup allowing for up to around 50% discount when negotiated in large orders. Enterprise gear also had a lifecycle of around 4.5 years while it still might technically work, there wouldn't be vendor support or replacements for them, and so enterprise customers are locked into perpetual planned obsolescence consumption cycles.
- r0ckarong 1y agoMy guess is someone somewhere is very satisfied that this data is now unrecoverable.
- gritzko 1y agoIn a world where data centers burn and cables get severed physically, the entire landscape of tradeoffs is different.
- fijiaarone 1y agoWhat info needed to be destroyed and who did it implicate?
- crmd 1y agoI would love to know how a fire of this magnitude could happen in a modern data center.
- AnimalMuppet 1y agoAllegedly from replacing batteries.
- esskay 1y agoOften poor planning or just lithium based batteries far too close to the physical servers. OVH's massive fire a couple of years ago in one of the most modern DC's at the time was a prime example of just how wrong it can go.
- bell-cot 1y agoAssume the PHB's who wouldn't spring for off-site backups (vs. excuses are "free") also wouldn't spring for fire walls, decently-trained staff, or other basics of physical security.
- filloooo 1y agoTheir decade-old NMC li-ion UPSs were placed 60cm away from the server racks.
- BonoboIO 1y agoEasy, some electronic fault and look at OVH with WOODEN FLOOR and bad management decisions. But of course the servers had automatic backups … in the datacenter in the same building. A few companies lost EVERYTHING and had to close, because of this.
- nullable_bool 1y agoI like to think that at least one worker was loafing on a project that was due the next day and there was no way it was going to get done. Their job was riding on it. They got drunk to embrace the doom that faces them, only to wake up with this news. Free to loaf another day!
- kupopuffs 1y agojust his luck
- mekoka 1y agoThis is wild. Wilder would be to see that the government runs the same after the loss.
- deleted 1y ago[deleted]
- WiggleGuy 1y agoI was in Korea during the Kakao fire incident and thought it was astounding that they had no failovers. However, I thought it'd be a wake up call! I guess not.
- HeavyStorm 1y agoWell I'll be. Backup is a discipline to not be taken lightly by any organization, specially a government. Fire? This is backup 101: files should be backed up and copies should be physically apart to avoid losing data. There are some in this threading pointing out that this would be handled by cloud providers. That bad - you can't hope for transparent backup, you need to actively have a discipline over it. My fear is that our profession has become very amateurish over the past decade and a lot of people are vulnerable to this kind of threat.
- creakingstairs 1y agoOne of the workers jumped off a building. [1] They say the person was not being investigated for the incident. But I can’t help but think he was a put under intense pressure to be scapegoat for how fucked up Korea can be in situations like this. To be some context on Korea IT scene, you get pretty good pay and benefits if you work for a big product company, but will be treated like dogshit inside subcontracting hell if you work anywhere else. [1] https://www.hani.co.kr/arti/society/society_general/1222145.html https://www.hani.co.kr/arti/society/society_general/1222145....
- deleted 1y ago[deleted]
- biglyburrito 1y ago"A source from the Ministry of the Interior and Safety said, “The G-Drive couldn’t have a backup system due to its large capacity” " :facepalm:
- deleted 1y ago[deleted]
- unit149 1y ago[dead]
- jiggawatts 1y agoI was the principal consultant at a subcontractor to a contractor for a large state government IT consolidation project, working on (among other things) the data centre design. This included the storage system. I noticed that someone had daisy-chained petabytes of disk through relatively slow ports and hadn’t enabled the site-to-site replication that they had the hardware for! They had the dark fibre, the long-range SFPs, they even licensed the HA replication feature from the storage array vendor. I figured that in a disaster just like this, the time to recover from the tape backups — assuming they were rotated off site, which might not have been the case — would have been six to eight weeks minimum, during which a huge chunk of the government would be down. A war might be less disruptive. I raised a stink and insisted that the drives be rearranged with higher bandwidth and that the site-to-site replication be turned on. I was a screamed at. I was called unprofessional. “Not a team player.” Several people tried to get me fired. At one point this all culminated in a meeting where the lead architect stood up in front of dozens of people and calmly told everyone to understand one critical aspect of his beautiful design: No hardware replication!!! (Remember: they had paid for hardware replication! The kit had arrived! The licenses were installed!) I was younger and brave enough to put my hand up and ask “why?” The screeched reply was: The on-prem architecture must be “cloud compatible”. To clarify: He thought that hardware-replicated data couldn’t be replicated to the cloud in the future. This was some of the dumbest shit I had ever heard in my life, but there you go: decision made. This. This is how disasters like the one in South Korea happen. In private organisations you get competent shouty people at the top insisting on a job done right. In government you get incompetent shouty people insisting that the job gets done wrong.
- Swenrekcah 1y ago> In private organisations you get competent shouty people at the top insisting on a job done right. In government you get incompetent shouty people insisting that the job gets done wrong. Great post and story but this conclusion is questionable. These kinds of incompetences or misaligned incentives absolutely happen in private organisations as well.
- jiggawatts 1y ago
- foofoo12 1y agoThis is extraordinarily loony shit. Someone designed a system like this without backups? Someone authorized it's use? Someone didn't scream and yell that this was bat and apeshit wacky level crazy? Since 2018? Christ almighty.
- hopelite 1y agoDoes anyone have an understanding of what the impact will be of this, i.e., what kind of government impact scale and type of data are we talking about here? Is this going to have a real impact in the near term? What kind of data are we’re talking about being permanently lost?
- spawarotti 1y agoThere are two types of people: those who do backups, and those who will do backups.
- mmaunder 1y agoAre we talking about actual portable thunderbolt 3 connected RAID 5 G-drive arrays with between 70 and 160TB of storage per array? We use that for film shoots to dump TB of raw footage. That G-Drive?? The math checks at 30GB for around 3000 users on a single RAID5 array. This would be truly hilarious if true.
- ChuckMcM 1y agoArticle comments aside, it is entirely unclear to me whether or not there was no backups. Certainly no "external" backups, but potentially "internal" backups. My thinking is that not actually allowing backups and forcing all data there creates a prime target for the PRK folks right? I've been in low level national defense meetings about security where things like "you cannot backup off site" are discussed but there are often fire vaults[1] on site which are designed to withstand destruction of the facility by explosive force (aka a bomb) or fire or flood Etc. That said, people do make bad calls, and this would be an epically bad one, if they really don't have any form of backup. [1] These days creating such a facility for archiving an exabyte of essentially write mostly data are quite feasible. See this paper from nearly 20 years ago: https://research.ibm.com/publications/ibm-intelligent-bricks-project-petabytes-and-beyond https://research.ibm.com/publications/ibm-intelligent-bricks...
- acchow 1y agoThey did have backups. But the backups were also destroyed in the same fire.
- derleyici 1y agoThen it's just incompetence. Even I have my backup server 100 km away from the master one.
- jedimastert 1y ago> My thinking is that not actually allowing backups and forcing all data there creates a prime target for the PRK folks right? It's funny that you mention that... https://phrack.org/issues/72/7_md#article https://phrack.org/issues/72/7_md#article
- ChuckMcM 1y agoOuch
- kwhitefoot 1y ago> there are often fire vaults[ Many years ago I was Unix sysadmin responsible for backups and that is exactly what we did. Once a week we rotated the backup tapes taking the oldest out of the fire safe and putting the newest in. The fire safe was in a different building. I thought that this was quite a normal practice.
- nowittyusername 1y agoI must say, at least for me personally when I hear about such levels of incompetence it rings alarm bells in my head making me think that maybe intentional malice was involved. Like someone higher up had set up the whole thing to happen in such a matter because there was a benefit to this happening we are unaware of. I think this belief maybe stems from lack of imagination on how really stupid humans can get.
- quantumsequoia 1y agoMost people overestimate the prevalence of malice, und underestimate the prevalence of incompetence
- deleted 1y ago[deleted]
- mliezun 1y agoWhat do you make of this? The guy who was in charge of restoring the system was found dead https://www.thestar.com.my/aseanplus/aseanplus-news/2025/10/03/official-responsible-for-restoration-of-south-korea039s-fire-hit-state-it-systems-found-dead https://www.thestar.com.my/aseanplus/aseanplus-news/2025/10/...
- BizarroLand 1y agoMy guess would be that either he felt it was such a monumental cockup that he had to off himself or his bosses thought it was such a monumental cockup that they had to off him.
- ajjahs 1y ago[dead]
- deleted 1y ago[deleted]
- vayup 1y agoA lot of folks are arguing that the real problem is that they refused to use US cloud providers. No, that's not the issue. It's a perfectly reasonable choice to build your own storage infrastructure if it is needed. But the problem is they sacrificed "Availability" in pursuit of security and privacy. Losing your data to natural and man-made disasters is one of the biggest risks facing any storage infrastructure. Any system that cannot protect your data against those should never be deployed. "The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for external backups." This is not a surprise to them. They had knowingly accepted the risk of infrastructure being destroyed by natural and man-made disasters. I mean, WTF!
- TulliusCicero 1y agoYeah, it's such a lame excuse to say "did not allow for external backups", as if that's a reasonable choice that they just couldn't work around. South Korea isn't some poor backwater, they have tech companies and expertise, that they were "unable" to do backups was an intentional choice.
- reed1234 1y agoDurability is more precise than availability in this context because it is about the data surviving (not avoiding downtime).
- john-tells-all 1y agoThis is literally comic. The plot of the live action comic book movie "Danger: Diabolik" [1] has a segment where the a country's tax records are destroyed, thus making it impossible for the government to collect taxes from its citizens. [1] https://en.wikipedia.org/wiki/Danger:_Diabolik https://en.wikipedia.org/wiki/Danger:_Diabolik
- 725686 1y agoIn my twenties I worked for a "company" in Mexico that was the official QNX ditribuitor for Mexico and LatAm. I guess the only reason was that Mexico City's Metro used QNX, and every year they bought a new license, I don't know why. We also did a couple of sales in Colombia I think, but was a complete shit show. We really just sent them the software by mail, and they had all sorts of issues getting it out of customs. I did get to go to a QNX training in Canada, which was really cool. Never got to use it though.
- throwaway2037 1y agoAt the very bottom of the article, I see this notice: > This article was originally written in Korean and translated by a bilingual reporter with the help of generative AI tools. It was then edited by a native English-speaking editor. All AI-assisted translations are reviewed and refined by our newsroom. I like that. It is direct and honest. I'm fine with people using LLMs for natural language related work, as long as they are transparent about it.
- BlueTemplar 1y agoIt's still worse than useless : https://www.bloodinthemachine.com/p/ai-killed-my-job-translators https://www.bloodinthemachine.com/p/ai-killed-my-job-transla...
- positron26 1y agoI really don't get this take where people try to downplay AI the most where it is obviously having the most impact. Sure. A billion people are supposed to go back to awful machine translation so that a few tens of thousands can have jobs that were already commodity.
- schrodinger 1y agoI have sympathy for those affected but this article is disingenuous. I speak Spanish and have just gone to 3 or 4 Spanish news sites, and passed their articles through to ChatGPT to translate "faithfully and literally, maintaining everything including the original tone." First it gave a "verbatim, literal English translation" and then asked me if I would like "a version that reads naturally in English (but still faithful to the tone and details), or do you want to keep this purely literal one?" Honestly, the English translation was perfect. I know Spanish, I knew the topic of the article and had read about it in the NYTimes and other English sources, and I am a native English speaker. It's sad, but you can't put the toothpaste back in the tube. LLMs can translate well, and the article saying otherwise is just not being intellectually honest.
- alsetmusic 1y ago
- Awesomedonut 1y agoNot even one redundant backup? That's unimaginable for me
- UltraSane 1y agoThis is amazingly incompetent because all the major enterprise storage arrays support automatic replication to remote arrays.
- hero4hire 1y agoIn 2025 data storage used by nation states, exposed to the internet, has no backups. No offsite backups. No onsite backups. No usb drives laying around unsecure in a closet. Nothing. What?
- anal_reactor 1y ago> no back-ups Top fucking kek. What were they expecting to happen? Like, really? What were they thinking?
- 3eb7988a1663 1y agoWhile I am sure a huge portion of valuable work will be lost, I am smirking thinking of management making a call, "So, if there is any shadow IT who has been running mirror databases of valuable infrastructure, we would have a no questions asked policy on sharing that right now". I know that I have had to keep informal copies of valuable systems because the real source of truth is continually patched,offline,churn,whatever.
- audiodude 1y agoReminds me when Toy Story 2 was deleted and they found the backups on an artist's laptop that was working from home.
- Blahagun 1y agoIt was on their SGI workstation that they lugged to home, but yeah, pretty much that's how they recovered most of the files. At the end they barely used the material.
- tropdrop 1y ago>artist technically, it was the supervising technical director. The only reason this happened (I don't think "working from home" was very common in 1999) was because she just had a baby! I love this story because it feels like good karma – management providing special accommodations for a new mom saves the show.
- hobofan 1y agoIf SK is anything similar to Germany or Japan in how they are digitizing their government processes, you'll probably be able to find paper printouts of all the data that was lost.
- ThePowerOfFuet 1y agoThe fun part will be finding them, figuring out their relevance, and re-digitizing them in a useful form.
- sgammon 1y ago> There is a cert and private key for rc.kt.co.kr, South Korea Telecom's Remote Control Service. It runs remote support backend from https://www.rsupport.com https://www.rsupport.com. Kim may have access to any company that Korea Telecom was providing remote support for.
- efitz 1y agoThe lack of backups makes my blood boil. However, from my own experience, I want to know more before I assign blame. The very first "computer guy" job I had starting in about 1990/1991, my mentor gave me a piece of advice that I remember to this day: "Your job is to make sure the backups are working; everything else is gravy." While I worked in that job, we outgrew the tape backup system we were using, so I started replicating critical data between our two sites (using 14400 bps Shiva NetModems), and every month I'd write a memo requesting a working backup system and explaining the situation. Business was too cheap to buy it. We had a hard drive failure on one of our servers, I requested permission to invalidate the drive's warranty because I was pretty sure it was a bad bearing; I got it working for a few weeks by opening the case and spinning the platter with my finger to get it started. I made sure a manager was present so that they could understand how wack the situation was- they bought me a new drive but not the extras that I asked for, in order to mirror. After I left that job, a friend of mine called me a month later and told me that they had a server failure and were trying to blame the lack of backups on me; fortunately my successor found my stack of memos.
- LorenPechtel 1y agoYeah. I've seen it. Had one very close call. The thieves took an awful lot of stuff, including the backups, had they taken the next box off the server room rack the company would have been destroyed. They stole one of our trucks (which probably means it was an inside job) and appear to have worked their way through the building, becoming more selective as they progressed. We are guessing they filled the truck and left. Did anything change? No.
- nicbou 1y ago> fortunately my successor found my stack of memos Those, ironically, were backed up
- lettergram 1y agoThe irony -- so not only was their system hacked ("hosted onsite"), but then it was also burned down onsite with no backups. In other words.. there was no point in the extra security of being onsite AND the risks of being onsite single failure point destroyed any evidence. Pretty much what I'd expect tbh, but no remote backup is insane.
- agnishom 1y agoThey were using a private service to manage public infrastructure? One developed by a foreign company?
- vimredo 1y agoThe G in G-Drive stands for Government, not Google. It tricked me too.
- bane 1y agoGoodness, I have over 100TB at home and it cost less than a two or three thousand dollars to put in place. That's like $25 per TB. > The stored data amounts to 858TB (terabytes), equivalent to 449.5 billion A4 sheets. No, the 858TB amounts to under $25k for the government of the 10th largest economy, of one of the most sophisticated countries on the planet, to put in place. Two of those would be less than the price of a new Hyundai Grandeur car. > “It’s daunting as eight years’ worth of work materials have completely disappeared.” So they're clocking in at around 100TB/year or 280GB a day. It's respectable, but not crazy. It's about 12GB/hr, doable with professional, server level hardware with backup moved over dedicated fiber to an offsite location. Multiply the price 10x and you can SSD the entire thing. Even with data sovereignty consideration demanding an entirely 100% home grown solution rather than turning to AWS or Azure, there's no excuse. But it's not like the cloud providers don't already have CSAP certification and local, in country, sovereign clouds [1] with multiple geographic locations in country [2] South Korea is full of granite mountains, maybe its time the government converts one into an offsite, redundant backup vault? 1 - https://erp.today/south-korea-microsoft-azure-first-hyperscaler-to-serve-korean-public-sector/ https://erp.today/south-korea-microsoft-azure-first-hypersca... 2 - https://learn.microsoft.com/en-us/azure/reliability/regions-list https://learn.microsoft.com/en-us/azure/reliability/regions-...
- jart 1y agoThe most sophisticated countries and companies are smart enough to use the least sophisticated backup methods. SK needs to backup their data to cassette tapes and tape libraries cost a bit more than that, but not much. Even if they boat their tapes over to an iron mountain in the US, I can't imagine the equipment and service fees are going to cost them more than a few hundred grand. They'll be spending more on the headcount to manage the thing.
- hedora 1y agoThe operational expenses of this stuff dwarfs the hardware cost. For the tape mountain, you need robots to confirm the tapes still work (mean time to detection of device failure and recovery are key for RAID durability computations). So, someone needs to constantly repair the robots or whatever. If I was being paid to manage that data set, I’d probably find two enterprise storage vendors, and stick two copies of the data set on them, each with primary secondary backup. Enterprise flash has been under a dollar a gigabyte for over a decade, so that’s under $1.7M per copy, amortized over five years. That’s $700K per year, and one of the four copies (at 3-4 sites) could be the primary store. (I can’t be bothered to look up current prices, but moore’s law says there have been six capacity doublings since then, and it still applies to flash and networking, so divide my estimate by 2^6 — so, ten-ish grand per year, with zero full time babysitters required).
- leentee 1y agoNo backup, no replica? Such a shame.
- fredsmith219 1y agoWow. Maybe backups would have been a good idea.
- BonoboIO 1y agoLook at OVH a few years … they had backups in the same datacenter. https://www.datacenterdynamics.com/en/news/ovhcloud-fire-report-sbg2-data-center-had-wooden-ceilings-no-extinguisher-and-no-power-cut-out/ https://www.datacenterdynamics.com/en/news/ovhcloud-fire-rep...
- yongjik 1y agoI see some comments about North Korean hacking, so I feel I need to clear up some misconceptions. First, (as you guys have seen) South Korea's IT security track record is not great. Many high-profile commercial sites have been hacked. If a government site was hacked by North Korea, it won't be the first, and while it would be another source of political bickering and finger-pointing, it's likely to blow over in a month. In fact, given that SK's president Lee started his term in June after his predecessor Yoon's disastrous attempt at overthrowing the constitution, Lee could easily frame this as a proof of the Yoon admin's incompetence. But deliberately setting fire on a government data center? Now that's a career ending move. If that's found out, someone's going to prison for the rest of their life. Someone should be really desperate to attempt that kind of thing. But what thing could be so horrible that they would rather risk everything to burn the evidence? Merely "we got hacked by North Korea" doesn't cut it. Which brings us to the method. A bunch of old lithium batteries, overdue for replacement, and predictably the job was sold to the lowest bidder - and the police knows the identity of everyone involved in the job and is questioning them as we speak. So if you are the evil perpetrator, either you bribed one of the lowest wage workers to start a fire (and the guy is being questioned right now), or you just hoped one of the age-old batteries would randomly start fire. Neither sounds like a good plan. Which brings us to the question "Why do people consider that plausible?" And that's a doozy. Did I mention that President Yoon almost started a coup and got kicked out? Among the countless stupid things he did, he somehow got hooked up on election conspiracy theories that say that South Korea's election commission was infiltrated by Chinese spies (along with major political parties, newspapers, courts, schools, and everything) and they cooked the numbers to make the (then incumbent) People's Power Party to lose congressional election of 2024. Of course, the theory breaks down the moment you look close. If Chinese spies had that much power, how come they let Yoon win his own election in 2022? Never mind that South Korea uses paper ballots and every ballot and every voting place is counted under the watch of representatives from multiple parties. To change numbers in one counting place, you'll have to bribe at least a dozen people. Good luck doing that at a national scale. But somehow that doesn't deter those devoted conspiracy theorists, and now there are millions of idiots in South Korea who shout "Yoon Again" and believe our lord savior Trump will come to Korea any day soon, smite Chinese spy Lee and communist Democratic Party from their seats, and restore Yoon at his rightful place at the presidential office. (Really, South Korea was fortunate that Yoon had the charisma of a wet sack of potatoes. If he were half as good as Trump, who knows what would have happened ...) So, if you listen to the news from South Korea, and somehow there's a lot of noise about Chinese masterminds controlling everything in South Korea ... well now you know what's going on.
- kepano 1y agoWhen I visited the National Museum of Korea in Seoul, one of my favorite parts was exploring the exhibit dedicated to the backing up state data — via calligraphy, letterpress, and stone carving. > "The Veritable Records of the Joseon Dynasty, sometimes called sillok (실록) for short, are state-compiled and published records, documenting the reigns of the kings of the Joseon dynasty in Korea. Kept from 1392 to 1865, they comprise 1,893 volumes and are thought to be the longest continual documentation of a single dynasty in the world." > "Beginning in 1445, they began creating three additional copies of the records, which they distributed at various locations around Korea for safekeeping." https://en.wikipedia.org/wiki/Veritable_Records_of_the_Joseon_Dynasty https://en.wikipedia.org/wiki/Veritable_Records_of_the_Joseo... After the Japanese and Qing invasions of Japan, King Hyeonjong (1659–1675) started a project to collect calligraphy works written by preceding Joseon kings and carve them into stone. It's somewhat surprising that these values didn't continue to persist in the Korean government.
- burnt-resistor 1y agoDR/BCP fail. The old adage companies that lose all of their data typically go out of business within 6 months I guess doesn't apply when it's the government. At a minimum, they could've stored the important bits like financial transactions, personnel/HR records, and asset inventory database backups to Tarsnap [0] and shoved the rest in encrypted tar backups to a couple of different providers like S3 Glacier and/or Box. Business impact analysis (BIA) is a straightforward way to assessing risks of probability of event * cost to recover from event = approximate budget for spending on mitigation. And, PSA: test your backups and DR/BCP runbooks periodically! 0. https://www.tarsnap.com https://www.tarsnap.com
- sbinnee 1y agoWhat a sad news as a Korean to see a post about Korea at the top of HN during one of the largest Korean holiday. I can share an anecdote how slow tech adoption is in Korea. It is not exactly about tech in public section but in private companies. I assume public section has slower adoption rate than private ones in general. Just about a year ago I had a couple of projects with insurance companies. I won't name them but they are the largest ones whose headquarters you can find in the very center of Seoul. They often called me in because I was setting up on-premise servers for the projects. Not to mention that it was hard to understand their choices of database architecture to plug it into the server I was setting up, their data team seemed just incompetent, not knowing what they were doing. The wildest thing I found was that most office workers seemed to be using windows 2000 to run their proprietary software. To be fair, I like software UIs with a lot of buttons and windows from that era. But alas, I didn't want to imagine myself connecting that legacy software to my then project service. It didn't go that far in the end.
- v7engine 1y agoDo South Korean companies prefer hosting data on their own servers instead of using Public cloud providers like Azure, AWS, GCP?
- sbinnee 1y agoYes and no. They used to prefer everything on premise. Many try to move towards cloud especially newer companies. Major cloud providers you mentioned are not the usual choices though (maybe aws is the most common). They do have data centers in Seoul and try to expand their markets for South Korea. But government offers generous incentives for using domestic cloud providers like NHN which was mentioned in the article or Naver cloud. Why does this work? Because Korean services rarely target global markets mainly due to language barrier. Domestic cloud usage is sufficient enough.
- bryanhogan 1y agoI think it's very interesting that Korea is probably the country with the fastest cultural adoption of new tech, e.g. #1 for ChatGPT, but on the other hand I can see as a web developer that new web tech is often adopteded at a very slow rate.
- deleted 1y ago[deleted]
- nwellinghoff 1y agoCould be incompetence. Highly likely. Or could be…suspect.
- marcus_holmes 1y agoIs it just me, or is this a massively better result than "1PB of government documents containing sensitive data about private individuals was exfiltrated to a hacker group and found for sale"? I applaud them for honouring their obligation to keep such data private. And encourage them to work on their backup procedures while continuing to honour that obligation.
- jeroenhd 1y agoA sibling comment links to a phrack page (https://phrack.org/issues/72/7_md https://phrack.org/issues/72/7_md) about North Korean infiltration in South Korean systems. The timing on that page and the fire make for a possible, though in my opinion wildly unlikely, scenario where either a saboteur started the fire when investigations were supposed to start, or (if you like hacking movies) that a UPS battery was rigged to cause a fire by the spies inside of the South Korean systems. It's possible that this is all just a coincidence, but the possibility that North Korea is trying to cover their tracks is there.
- awesomeusername 1y agoI'm CTO of a TINY company, with pretty much exactly half this data. I run all storage and offside backups personally, because I can't afford a full time sysadmin yet. And the cost of everything is PAIN to us. If our building burned down we would lose data, but only the data we are Ok with losing in a fire. I'd love to know the real reason. It's not some useless tech... it's politics, surely.
- dbuser99 1y agoSometimes it is convenient that there are no backups. Just saying…
- Jean-Papoulos 1y ago>the G-Drive’s structure did not allow for external backups That should be classified as willful sabotage. Someone looked at the cost line for having backups in another location and slashed that budget to make numbers look good.
- pammf 1y agoReal reason is humans are way too optimistic in planning and, for some reason, tend to overlook even more rare, but catastrophic risks. I’m almost sure that the system had some sort of local replication and versioning that was enough to deal with occasional deletions, rollbacks, and single non-widespread hardware failures, so only the very catastrophic scenario of losing all servers at the same time (that for sure wouldn’t happen anytime soon) was uncovered.
- mrweasel 1y agoAt a previous job I was not allowed to do disaster planning with customers, after I told one of them that it was entirely possible to take out both our datacenters with one plane crash. The two locations where a "safe" distance apart, but where also located fairly close the approach of an airport, and a crashing passenger jet is big enough to take out both buildings. Apparently I plan for the rather rare catastrophes, and not those customers care about day to day.
- mewpmewp2 1y agoHowever it's also possible that an asteroid could destroy everything or a nuclear war.
- lordnacho 1y agoBut it's extra surprising, because South Korea is a country where every young man is conscripted due to the threat of war with the north. If the conflict is serious enough for that, why hasn't someone thought about losing all the government data in a single artillery strike?
- athrowaway3z 1y agoIt would be wise for governments to define "backup" as something that is at least 1km away.
- tylervigen 1y agoProbably farther than that, right? Plenty of natural disasters, including floods and wildfires, can affect an area larger than 1 km.
- anticensor 1y agoFarther than 100km..
- ptojr 1y agoI wouldn't be surprised if someone caused this intentionally.
- Ylpertnodi 1y ago> I wouldn't be surprised if someone caused this intentionally. What, no backup(s) set up? Hmmm, possibly. But, they're'd be a paper trail. Imagine all the scrabbling going on right now - people desperately starting to cover their arses. But chances are, what they need has just burnt down, with no backups.
- Lumoscore 1y agoWow. That is genuinely one of the most terrifying headlines I've read all year. Seriously, "no backups available" for a national government's main cloud storage? That’s not a simple IT oversight; that’s an epic, unforgivable institutional mistake. It completely exposes the biggest fear everyone in tech has: putting all the eggs in one big physical basket. I mean, we all know the rule: if it exists in only one place, it doesn't really exist. If your phone breaks, you still have your photos on a different server, right? Now imagine that basic, common-sense rule being ignored for a country’s central data. The fire itself is a disaster, but the real catastrophe is the planning failure. They spent millions on a complex cloud system, but they skipped the $5 solution: replicating the data somewhere else—like in a different city, or even just another building across town. Years of official work, policy documents, and data—just gone, literally up in smoke, because they violated the most fundamental rule of data management. This is a massive, expensive, painful lesson for every government and company in the world: your fancy cloud setup is worthless if your disaster recovery plan is just "hope the building doesn't burn down." It’s an infrastructure nightmare.
- hooskerdu 1y agoDoes a half-decent job of breaking down how things were affected https://youtu.be/j454KF26IWw https://youtu.be/j454KF26IWw
- NinjaTrance 1y agoThe easy solution would be to use something like Amazon S3 to store documents as objects and let them worry about backup; but governments are worried (and rightly so) about the US government spying on them. Thus, the not-so-easy-but-arguably-better solution would be to self-host an open source S3-compatible object storage solution. Are there any good open source alternatives to S3?
- jankal 1y agoWhen you wish the S. to be an N.
- clayliu 1y ago[dead]
- ken47 1y agoCoincidence is God’s way of remaining anonymous.
- admiralrohan 1y agoIs there any solution to these kind of issues other than having multiple backups and praying that not all of them will caught fire at the same time?
- rstuart4133 1y agoBackups are best thought of as a multi dimensional problem, as in, they can be connected in many dimensions. Destroy a backup, and all those in the same dimension are also destroyed. This means you must have to have redundancy in many dimensions. That all sounds a bit abstract, so ... One dimension is two backups can be close in space (ie, physically close, as happened here). Ergo backups must be physically separated. You've heard RAID can't be a backup? Well it sort of can, and the two drives can be physically separated in space. But they are connected in another dimension - time, as in they reflect the data at the same instant in time. So if you have a software failure that corrupts all copies, your backups are toast as you can't go back to a previous point in time to recover. Another dimension is administrative control. Google Drive for example will backup your stuff, and separate it in space and time. But they are connected by who controls them. If you don't pay the bill or piss Google off, you've lost all your backups. I swear every week I see a headline saying someone lost their data this way. Then backups can be all connected to you via one internet link, or connected to one electrical grid, or even one country that goes rogue. All of those are what I called dimensions, that you have to ensure your backups are held at a different location in each dimension. Sorry, that didn't answer your question. The answer no. It's always possible all copies could be wiped out at the same time. You are always relying on luck, and perhaps prayer if you think that helps your luck.
- admiralrohan 1y agoInteresting way to explain this through multiple dimensions angle.
- OJFord 1y ago> The scale of damage varies by agency. [...] The Office for Government Policy Coordination, which used the platform less extensively, Amazing
- alwahi 1y agoGovernment fires are never a mistake
- phatfish 1y ago"The stored data amounts to 858TB (terabytes), equivalent to 449.5 billion A4 sheets" Just so we can all visualise this in an understandable way, if laid end-to-end how many times round the world would the A4 sheets go? And what is their total area in football fields?
- porkbrain 1y agoIf you stacked them they would be about fifty thousand Popocatépetls high, give or take a few zeroes. UPDATE: as sibling pointed out indirectly, it's eight thousand Popocatépetls [0]. [0]: https://www.wolframalpha.com/input?i=449.5+*10%5E9+*+%28thickness+of+a+sheet+of+paper%29+%2F+%28height+of+Popocat%C3%A9petl%29 https://www.wolframalpha.com/input?i=449.5+*10%5E9+*+%28thic...
- akpa1 1y ago190,813,414 and a bit times round the equator if you place them long edge to long edge
- panda-giddiness 1y agoAttached end-to-end, they'd extend almost from the Earth to the Sun [1]. Placed in a grid, they'd cover an area larger than Wales [2]. Piled on top of each other, they'd reach a tenth the distance to the moon [3]. --- [1] https://www.wolframalpha.com/input?i=449.5+*10%5E9+*+%28length+of+an+A4+sheet%29+%2F+1+AU https://www.wolframalpha.com/input?i=449.5+*10%5E9+*+%28leng... [2] https://www.wolframalpha.com/input?i=449.5+*10%5E9+*+%28area+of+an+A4+sheet%29 https://www.wolframalpha.com/input?i=449.5+*10%5E9+*+%28area... [3] https://www.wolframalpha.com/input?i=449.5+*10%5E9+*+%28thickness+of+a+sheet+of+paper%29+%2F+%28distance+to+moon%29 https://www.wolframalpha.com/input?i=449.5+*10%5E9+*+%28thic...
- nonethewiser 1y agoI am shocked that 1 and 2 are both true. I would have guessed 1 would have implied a much larger area than Wales.
- 1y ago
- hamdouni 1y agoIt is not cloud storage if it's not resilient... It's just remote storage.
- alexnewman 1y agoIndistinguishable from crime.
- heisenbit 1y agoIt is very unlikely that the low performance would have prevented any backup. This was slow changing data. Here the real inability to a good, solid backup was taken as an excuse to do anything at all.
- ZiiS 1y agoAre we actually sure they didn't do due diligence? This is the individual's work files of civil servants. These will overwhelmingly be temporary documents they were legally obliged to delete at some point in the last 8 years. Any official filings or communications would have been to systems of record that were not effected. This is more that a very large fire, probably unlucky for once a decade, caused civil servants to lose hours of work in files they were working on. A perfect system could have obviously prevented this and ensured availability, but not without cost.
- rtkwe 1y agoWell it's really in the cloud(s) now! /s No offsite backups is a real sin, sounds like a classic case where the money controllers thought 'cloud' automatically meant AWS level redundant cloud and instead they had a fancy centralized datacenter with insufficient backups.
- ionwake 1y agoIts bizarre how easy it to make smart people on HN just assume people who are doing something weird are just low IQ. Its almost weirdly a personality trait that a trained programmer just goes around believing everyone around him doesnt understand what way the wind blows. Government installation for backups for a government ruled by a weird religious sect, have no offsite backups, it goes up in flames? Well clearly they were not smart enough to understand what an off-site backup is. Its like wtf guys? Now dont get me wrong, occams razor, they tried to save a few bucks, it all went Pete tong, but cmon, carelessness , chance, sure, but I doubt its all down to stupidity.
- voidhorse 1y agoIt's a common problem in any field that presumably revolves around intellect, since supposedly being smarter gets you further (it may, but it is not enough on its lonesome). People, in general, severely overestimate their own intelligence and grossly underestimate the intelligence of others. Consider for a moment that most of the geniuses on hacker news are not even smart enough to wonder whether or not something like IQ is actually a meaningful or appropriate way to measure intelligence, examine the history of this notion, question what precisely it is we mean by that term, how its use can vary with context, etc. etc.
- ionwake 1y agoIs there a better word/assessment for "ability"? Just wondering what it would be, just "success" in a domain? I agree with you just wondering
- libria 1y agoYeah, all this chatter about technologies and processes that could have saved this: you don't think someone in all of Korean government knew about that? The problem is more likely culture, hierarchy or corruption. Guaranteed several principal security architects have been raising the alarm on this internally along with much safer, redundant, secure alternatives that came with an increased cost. And decision makers who had a higher rank/social/networking advantage shot it down. Maybe the original storage designer was still entrenched there and sabotaging all other proposals out of pride. Or there's an unspoken business relationship with the another department providing resources for that data center that generates kickbacks. Assuming nobody knows how to do an offsite backup or is plain ignorant of risk over there is arrogant.
- dudeinjapan 1y agoPeople keep pointing the finger at North Korea but personally I suspect a Protoss High Templar causing battery overcharge is more plausible.
- dangoodmanUT 1y agoTWO IS ONE ONE IS NONE
- abtinf 1y agoJim Hacker: How am I going to explain the missing documents to The Mail? Sir Humphrey: Well, this is what we normally do in circumstances like these. Jim Hacker: (reading) This file contains the complete set of papers, except for a number of secret documents, a few others which are part of still active files, some correspondence lost in the floods of 1967… Was 1967 a particularly bad winter? Sir Humphrey: No, a marvelous winter. We lost no end of embarrassing files.
- eranation 1y agoTo those wondering where it's from: https://www.imdb.com/title/tt0751825/quotes/?item=qt0238072 https://www.imdb.com/title/tt0751825/quotes/?item=qt0238072
- deleted 1y ago[deleted]
- deleted 1y ago[deleted]
- monster_truck 1y agoOne of the lessons I learned from my Network Administration teacher was that if you're ultimately responsible for it and they say no backups? You tack on the hours required to do it yourself (this includes the time you must spend actually restoring from the backups to verify integrity, anything less can not be trusted). You keep one copy in your safe, and another copy in a safety deposit box at the bank. Nobody ever has to know. It is inevitable that you will save your own ass, and theirs too. Shit happens.
- commandlinefan 1y agoI have full confidence that management will learn nothing from this object lesson.
- cindyllm 1y ago[dead]
- wigster 1y agoThat's not a cloud. That is smoke
- syngrog66 1y agosounds like will become textbook case lesson about backups and disaster planning
- ratelimitsteve 1y agoif you love it, make a copy of it this is the kind of thing that is so fundamental to IT that not doing it is at best negligence and at worst intentional malpractice. There is simply no situation that justifies not having backups and I think it might be worth assuming intentionality here, at least for purposes of investigation. It looks like an accident but someone (perhaps several someones, somefew if you will) made a series of shriekingly bad decisions in order to put themselves in a precarious place where an accident could have an effect like this.
- saltyoldman 1y ago> A firefighter cools down burnt batteries at the National Information Resources Service (NIRS) in Daejeon on Sept. 27. [YONHAP] New caption: > A firefighter wants to see the cool explosive reaction between water and lithium at the National Information Resources Service (NIRS) in Daejeon on Sept. 27. [YONHAP]
- calebm 1y agoMr Robot was here?
- ryanmcbride 1y agoHere I was self conscious about my homelab setup and turns out I was already way ahead of the second most technologically advanced nation in the world!
- achow 1y agoA little more informative source: https://www.datacenterdynamics.com/en/news/858tb-of-government-data-may-be-lost-for-good-after-south-korea-data-center-fire/ https://www.datacenterdynamics.com/en/news/858tb-of-governme... - G-drive stands for Government Drive - The incident was caused due to Lithium battery fire - The drive was of 858TB capacity - No backup because “The G-Drive couldn’t have a backup system due to its large capacity” (!!)
- ycombinatrix 1y agoGuess they'll have to ask China for their backup.
- 4WIW 1y agoThe board of directors should now fire the management over such as gross mismanagement. Then, the board of directors should be fired for not proactively requiring backups.
- bornfreddy 1y agoIs it possible that the fire was started by malicious software, for example by somehow gaining control of UPS batteries' controllers or something similar?
- polynomial 1y agoInsisting on having a SPF (single point of failure) for... reasons.
- bvan 1y agoHow could you even define that as a ‘cloud’? Sounds like good old client-server on a single premise, and no backup whatsoever. Can’t have had very secure systems either.. perhaps they can buy back some of the data off the dark web.. or their next-door neighbor.
- garfieldnate 1y agoI know Korea is a fast-changing place, but while I was there I was taught and often observed that the value of "ppalli ppalli" (hurry hurry) was often applied to mean that a job was better done quickly than right, with predictably shoddy results. Obviously I have no insight into what happened here, but I can easily imagine a group of very hurried engineers feeling the pressure to just be done with their G-Drive tasks and move on to other suddenly urgent things. It's easy to put off preparation for something you don't feel will ever come. I'm going to check all the smoke detectors in my house tomorrow :D
- stefek99 1y agoEach government should run a drill backup exercise.
- bigjobby 1y agoThis is a great fear of mine. I have data backups of backups. A 2 year project is coming to a close soon and I'll be able to relax again. Bring back paper printouts.
- ninjaa 1y agoDon't call it a coverup
- elgolem89 1y agoIn Latin America, this is the normal way to erase evidence of corruption...