9 ms·
Oracle Wallet Master Key Lost
- technion 7y agoI've been in a number of these situations where the basic workflow was "encryption = good" with no consideration for these scenarios. I've seen several organisations decide they need to encrypt backups. The first thing I ask is "where is that key saved" and the answer has always been "in a file, written to those backups". It's amazing how many times "encryption" leaves someone one ransomware away from losing everything.
- dpwm 7y agoEncrypting backups is more-or-less essential for many types of data – but you absolutely need to know what is at stake (complete data loss) and how to mitigate that (redundant hard-copies of keys stored securely both on- and off-site).
- LoSboccacc 7y ago> stored securely print the keys and give them to the cfo to store as asset, they have the systems in place to keep paper around for decades.
- newnewpdro 7y agoDo you really want finance involved in operational chores as mundane as restoring backups? I see no reason for this, whoever is head of the department responsible for disaster recovery should just stick a backup printout of the keys in a local safe deposit box and get the right names on the account.
- LoSboccacc 7y agoand who has the keys to that deposit? it's key all the way down. unless you are finance. never saw finance having their document storage delegated to a third party. it's their nature. that and legal, but since the master key is quite like a title to your data, finance made more sense.
- Xylakant 7y agoWhy not hand a printout to finance? It’s a last-ditch safety anchor in case everything else fails. Keep another copy for day-to-day operations in digital form. Nobody wants to type a 4096 bit key for every backup restore. But if millions are at stake, motivation to do so will rise tremendously.
- newnewpdro 7y agoI already said it's a backup copy of the key, obviously nobody is expected to type in from a printout in day-to-day operations. Our experiences must differ substantially here. I've worked at a few startups over the past two decades, and I would not trust any of the finance people from any of those companies to practice good operational security or be reliably available as a cog in a critical real-time operational path. They were consistently the kinds of people who would treat a locked filing cabinet behind a locked office door as the height of security, and in many cases left important private documents like checks revealing salaries completely insecure, face down on recipients unattended desks. They have often been the staff least reliably in the office, and least accessible at any given moment - especially after hours or on weekends. In my view it's just exposing the key to more risk of getting left out or forgotten in a poorly secured, often unattended filing cabinet behind trivially picked locks. Just to make it available through someone frequently absent on vacation. Good thing the security will likely suck, because you'll probably have to pick those locks when you need the copy and the CFO is skiing down a mountain. This is an issue for the department in charge of performing backups and restores. They will already have access to the key to do the job, and they need to be accessible in times of crisis. Why wouldn't the COO be the right person if operations is in charge of disaster recovery? It's their ass on the line when they can't do a restore on the weekend. The CFO and the entire finance department has zero expectation of being available after hours or on holidays.
- dagw 7y agoDo you really want finance involved in operational chores as mundane as restoring backups? Not as a day to day occurrence. But as a last line of defence, absolutely. No other department has better routines in place to keep papers safe and secure for years through rounds of layoffs, restructurings, buyouts and everything else a company might go through.
- mamon 7y agoWhy not just burn them on the CD? This way they are safe from hacker's attacks, and still in the digital form. There are special types of CDs that will last decades. There even is such thing as "1000 years dvd" :)
- kevin_thibedeau 7y agoThose special optical discs are traditionally pressed and gold sputtered. Burnable discs are not suitable for long term archives.
- gtyras2mrs 7y agoM-DISC claims a 1000 year lifespan. LG has some consumer writers that support M-DISC I believe. There are Sony BD-XL 128GB discs that claim 50 years of shelf life - however I'm not sure if they are suitable for archives. However as long as they are regularly tested - they could be decent enough for cold-storage. https://www.reddit.com/r/DataHoarder/comments/ataj1o/upping_the_optical_ante_sonys_128gb_quad_layer/ https://www.reddit.com/r/DataHoarder/comments/ataj1o/upping_...
- kakwa_ 7y agoPaper is proven to last a few hundred years without issues, if stored properly, and we know all the kinks of storing it properly. It's easy to read, contrary to a DVD (you need a DVD drive). It's also cheap. Why make things complex when they could be simple.
- 4ad 7y agoThe comments are terrible, blaming Oracle (I guess that's trendy these days), some people even advise suing Oracle. This is a system administration failure, not an Oracle failure. Basically they didn't test their recovery strategy.
- detaro 7y agoClearly something Oracle-side fucked up here too, so it's also an Oracle failure. Although the "support talked us through..." bits seem like a sign of "should probably have gone for something hosted", but depending on the contract could also spin to "so clearly Oracle must have told us something wrong".
- bhaak 7y agoThe key phrase in the posting are "Oracle is stumped". This is way more than enough to turn this over to lawyers and let the court decide who is to blame. Yes, the company could have checked their recovery strategy but if Oracle claims the key gets written to disk and it wasn't, there is a good chance for at least some compensation from Oracle.
- mmcgaha 7y agoThere must be something that I don't understand fully about this because It would see odd to me that data would be encrypted at rest, and at the same time have the plain text key available to decrypt the data.
- bhaak 7y agoI'm not familiar with the system in question but it is not stated if reading the key would need a password or not. But being a cloud service, it would be odd if the encryption key would be available without any interaction by the owners.
- bloat 7y agoHe says he has the password for the key wallet. But the wallet is empty.
- tomxor 7y agoAlways make multiple hard copies of encryption keys, it's the one thing your mirrored incremental offsite backup will not help you with.
- josu 7y agoThe problem is the wallet was empty.
- NotSammyHagar 7y agoBut you need more than one wallet. Just like you need more than one house key, in case you lose one.
- tomxor 7y agoThat would have been noticed if someone had at least attempted to make hard copies immediately. The second I encrypt a DB (which i've only done once so far, so i'm far from an expert), I get a copy of, verify and duplicate the key, otherwise all backups from that point onward will be useless... The point you add encryption and fuck up the keys is the point you are able to recover and try again, instead of waiting for days, weeks or months like these guys - you check it immediately - if you didn't get a copy of the key or it's invalid, that's ok, your old unencrypted backup isn't out of date yet.
- crispyambulance 7y agoI don't do this type of work, but come on, don't people practice doing such recoveries before they commit to such a system? Wouldn't performing dry-run with a checklist of actions have kept this from happening or at least alerted them to the deficiencies?
- marktangotango 7y agoHey man, it’s agile, cloud, devops; we don’t value sys admins, dbas, or business analyst! We hire inexperienced devs and expect them to do it all! Experience? Bah! To expensive! This is the world we’ve got today.
- zaphar 7y agoIf you read the thread it wasn't agile devops who caused the problem. It was a DBA following specific instructions from Oracle support. Which is like the polar opposites of what you describe. This company was screwed as soon as they did business with Oracle. A company whose entire business model is making things as inscrutable as possible so you'll pay them more money in support contracts.
- Jare 7y agoOracle never told them company to NOT test the backups. That was the company's own doing (and demise). So the question is, why did the company not think of testing their backups, and generally doing a full validation of the system before committing to it for months and millions of $ in business? What sort of workflows, processes and culture in the company allowed this to happen? Probably the cheap sort.
- marktangotango 7y agoFor those of us who reject the new ui https://old.reddit.com/r/sysadmin/comments/bn38pp/oracle_wallet_master_key_lost/ https://old.reddit.com/r/sysadmin/comments/bn38pp/oracle_wal...
- _asummers 7y agoMy worry is that this trick will go away soon. I really dislike the default Reddit UI.
- monsieurbanana 7y agoI hope that this trick will go away. I'd be much more productive.
- willio58 7y agoIt will, so why not start now? No more excuses!
- jcims 7y agoYou can also set it in preferences if you have a user account
- pjmlp 7y agoReddit tends to force the new UI, even with it disabled on preferences, with a hint of one wants to have another go at it.
- electrotype 7y agohttps://addons.mozilla.org/en-CA/firefox/addon/old-reddit-redirect/ https://addons.mozilla.org/en-CA/firefox/addon/old-reddit-re...
- perlgeek 7y agoThis once again reinforces the point that a backup is worthless if you haven't tested restore.
- cptskippy 7y agoThat's one of those idioms that rings hollow until it rings very true.
- SlowRobotAhead 7y ago“If you haven’t tested your backups, you don’t have backups.” LOTS of people don’t have backups.
- nemothekid 7y agoFrom the thread: >It's Oracle cloud - you can't really test them without restoring a snapshot (overwriting your current data) Jesus, if thats true, what a shit product. AWS RDS doesn't even let you restore a backup to an already running database, it always creates a new one.
- fauigerzigerk 7y agoRestore would have been successful while the key was still in memory. You could say if that happens it's a badly designed test, and I agree. But it goes to show that testing entire infrastructures partly run by service providers is not easy.
- CodeWriter23 7y agoRestore to the same system isn’t actually a restore test.
- deleted 7y ago[deleted]
- rwbhn 7y agoI think about it this way: you don't need a backup system - you need a restore system.
- Cieplak 7y agoIf the key was in RAM, they might be able to find it in a core dump or in swap memory. Obviously a proper key mgmt system should never let that key be written to disk without encrypting it first with a KEK, but that farther up their hierarchy of needs.
- vbezhenar 7y agoHow can you realistically find it? Probably requires writing very sophisticated software with questionable legal grounds. It might cost a lot in the end without guaranteed results.
- SlowRobotAhead 7y agoSo I did the math quick, but say you had 256GB of ram, and the key was in there somewhere (pretending it’s nicely lined up in some known endian and format), assuming you can visually cut 50% of RAM out by excluding areas the key definitely isn’t in (we know they key doesn’t have 4 repeating chars or is even 1/4 all zero), if you just cycle through each keysize, you need to be in the 10,000 guesses per second to spend less than a year on it. I have no idea the keysize or speed that is practical. But it’s hardly impossible to just dictionary attack using your old RAM as the “list”.
- jnwatson 7y agoYou could significantly narrow it down by only trying sequences of very high entropy.
- SlowRobotAhead 7y agoThat’s a much better way to describe what I was getting at.
- weinzierl 7y agoIf the keys where just plainly kept in RAM they should be easy to find in the dump. The reason is that key material has certain characteristics (high entropy) that make it easy to be found. Tools that can do this are freely available.
- village-idiot 7y agoFor some reason I read this as the Oracle Master Key was lost, and I was expecting much more fireworks.
- nytesky 7y agoI almost feel like encryption systems like this need physical keys. They can be secured in safe, maybe create 2 even.
- vbezhenar 7y agoPhysical keys can break. Just use password, print it and store few copies in different places.
- dfrage 7y agoExactly. Print out the whole thing, doesn't matter how many characters if you're trying to recover from something this drastic, and follow the general backup rule of "If you have 1 copy, you have 0, if you have 2, you have 1...." Put copies in different flood plains, ideally different parts of your country or the world. While its not always possible, it's a good idea to use at least two completely different backup methods, avoiding single points of failure like in this case. For example, use a logical volume manager to make frozen copies of your filesystems, and back those up using something low level.
- williesleg 7y agoOracle doing the needful once again.
- LoSboccacc 7y agoI don't know the tech involved so I'll ask as future reference: was it possible, before the reboot, to have the database write the key out, or was this database doomed forever once the key went missing?
- scarmig 7y agoIf the database was using the key to write encrypted data, it had the key in memory at least.
- bufferoverflow 7y agoBut not necessarily in one place, we don't know how Oracle stores encryption keys in RAM. So even if you magically have the whole RAM dump after the reboot, it wouldn't be a trivial task to find the key.
- gdsdfe 7y agoIt's interesting to read the comments on anything related to Oracle, the general sentiment is people hate it ... a lot! I wonder how long it will stay in business?
- snarfy 7y agoThe US government is their largest customer. Considering the glacial pace of the government, I suspect it will be a really long time.
- curiousgal 7y ago3rd world governments as well because they don't know any better.
- hotsauceror 7y agoOracle DB is deeply embedded in some mission critical "octopus" systems that touch large swaths of an enterprise (PeopleSoft, SAP, TIBCO ESB). It's disruptive, expensive, and time consuming to rip out the guts of your enterprise, particularly if you're not an IT business. Inertia is a hell of a drug.
- bec123 7y agoKey Management.
- justinclift 7y agoWonder how many other Oracle cloud environments will turn out to be bitten by the same bug?
- neop1x 7y agoSo the data was not encrypted and you thought it was? And you pay for this product? Aha...