7 ms·

The comments are terrible, blaming Oracle (I guess that's trendy these days), some people even advise suing Oracle. This is a system administration failure, no

by 4ad 7y ago

The comments are terrible, blaming Oracle (I guess that's trendy these days), some people even advise suing Oracle. This is a system administration failure, not an Oracle failure. Basically they didn't test their recovery strategy.

detaro 7y ago
Clearly something Oracle-side fucked up here too, so it's also an Oracle failure. Although the "support talked us through..." bits seem like a sign of "should probably have gone for something hosted", but depending on the contract could also spin to "so clearly Oracle must have told us something wrong".
bhaak 7y ago
The key phrase in the posting are "Oracle is stumped". This is way more than enough to turn this over to lawyers and let the court decide who is to blame. Yes, the company could have checked their recovery strategy but if Oracle claims the key gets written to disk and it wasn't, there is a good chance for at least some compensation from Oracle.
- mmcgaha 7y ago
  There must be something that I don't understand fully about this because It would see odd to me that data would be encrypted at rest, and at the same time have the plain text key available to decrypt the data.
  - bhaak 7y ago
    I'm not familiar with the system in question but it is not stated if reading the key would need a password or not. But being a cloud service, it would be odd if the encryption key would be available without any interaction by the owners.
  - bloat 7y ago
    He says he has the password for the key wallet. But the wallet is empty.
lykr0n 7y ago
If I pay you for a service, say hosting a database and management, that says it's encrypted at rest. Your system fails to complete the required steps for this process- namely fails to write the decryption key to a place where it can be recovered, It's your fault. Not the customers.
rtempaccount1 7y ago
I think some of that comes from this occurring in Oracle's cloud environment. It's not an on-prem setup where the customer has full control to manage/test backups, it's at least some level of managed service from Oracle. So at the least the commands used should have offered/enforced creation of a key backup as part of the encryption process (in the way that bitlocker does for Windows disk encryption) to reduce the risk of something like this happening.
jodrellblank 7y ago
From the comments, Oracle told them to enable encryption to solve a problem with backups. They did, and they kept the master key to the Oracle wallet. The Oracle wallet is read only, they could never put something in it or fail to put something in it. The encryption system could and should have put encryption keys in it, but didn't. That's the Oracle failure. "Enabling Oracle encryption according to their instructions, failed to do what it should, and we lost access to all our servers and all our backups because of it". There's also a separate failure of administration on the client side of not practising working with encryption before using it in production, and not testing recovery and noticing the wallet was empty and the backups were unusable before it became a disaster, but the fact that it did become a disaster is a failure of Oracle's system.
- gregd 7y ago
  I'm not so sure why everyone is on-board with blaming Oracle out of the gate. From one of the OP's comments, "Oracle support told him, step by step, to encrypt the database, which happened ages ago and is the root of all of this." That wallet is the responsibility of whomever created it. Just because it's empty now, doesn't mean it's always been empty. In addition to that, what if the person who initially encrypted everything with TDE, used a local only wallet? This would explain why the ewallet.p12 file is empty. Oracle recommends that you store your master key in the ewallet.p12 file and not the cwallet.sso file, which is the auto-open file, but only on the workstation it was created on. There could have been multiple failures here with multiple parties and not just Oracle.
- hotsauceror 7y ago
  I'd be very curious to know what drove that decision to recommend enabling encryption in the first place. That seems like an odd way to "solve a problem with backups", particularly given the huge risks inherent to irresponsible key management. It almost sounds like their original support engineer proposed this as a fix during a ticket. Normally something like this is something you'd plan out, test, and do risk analysis with the business rather than have your on-call deploy it in the middle of the night because John Q. Engineer said you should.
icedchai 7y ago
Sounds like they are using a managed database service. If that is the case, there is no system to administer and Oracle is at fault. Welcome to the cloud.
vbezhenar 7y ago
Yep. Probably Oracle has standard no warranties and obligations clause in their license.
djsumdog 7y ago
This is Oracle's hosted solution (like Amazon RDS) and their software failed. The OP even mentioned how the backups of their wallet were empty and this was never flagged somehow. If this was something they ran locally, yet it'd probably be their fault. But this is a hosted solution with a major bug. I hope some Oracle people are searching through seeing how many other customers have this empty wallet bug, and fixing it now.
- justinclift 7y ago
  Thinking it through, if they find other wallets with this problem, I wonder what the fix is? They could generate a new key + wallet, which could probably then be made to work for the running instance. All the backups though, would probably need to be opened with the current (missing) key+wallet in order to re-encrypt them. Might be doable with the current key+wallet that's in memory, but also it might not be. eg might not be able to change the software on the running system, in which case ouch... places that need historical backups (eg for legal purposes) would be in bad place.

by 4ad 7y ago

detaro 7y ago

Clearly something Oracle-side fucked up here too, so it's also an Oracle failure. Although the "support talked us through..." bits seem like a sign of "should probably have gone for something hosted", but depending on the contract could also spin to "so clearly Oracle must have told us something wrong".

bhaak 7y ago

The key phrase in the posting are "Oracle is stumped". This is way more than enough to turn this over to lawyers and let the court decide who is to blame. Yes, the company could have checked their recovery strategy but if Oracle claims the key gets written to disk and it wasn't, there is a good chance for at least some compensation from Oracle.

mmcgaha 7y ago
There must be something that I don't understand fully about this because It would see odd to me that data would be encrypted at rest, and at the same time have the plain text key available to decrypt the data.
- bhaak 7y ago
  I'm not familiar with the system in question but it is not stated if reading the key would need a password or not. But being a cloud service, it would be odd if the encryption key would be available without any interaction by the owners.
- bloat 7y ago
  He says he has the password for the key wallet. But the wallet is empty.

lykr0n 7y ago

If I pay you for a service, say hosting a database and management, that says it's encrypted at rest. Your system fails to complete the required steps for this process- namely fails to write the decryption key to a place where it can be recovered, It's your fault. Not the customers.

rtempaccount1 7y ago

I think some of that comes from this occurring in Oracle's cloud environment. It's not an on-prem setup where the customer has full control to manage/test backups, it's at least some level of managed service from Oracle. So at the least the commands used should have offered/enforced creation of a key backup as part of the encryption process (in the way that bitlocker does for Windows disk encryption) to reduce the risk of something like this happening.

jodrellblank 7y ago

From the comments, Oracle told them to enable encryption to solve a problem with backups. They did, and they kept the master key to the Oracle wallet. The Oracle wallet is read only, they could never put something in it or fail to put something in it. The encryption system could and should have put encryption keys in it, but didn't. That's the Oracle failure. "Enabling Oracle encryption according to their instructions, failed to do what it should, and we lost access to all our servers and all our backups because of it". There's also a separate failure of administration on the client side of not practising working with encryption before using it in production, and not testing recovery and noticing the wallet was empty and the backups were unusable before it became a disaster, but the fact that it did become a disaster is a failure of Oracle's system.

gregd 7y ago
I'm not so sure why everyone is on-board with blaming Oracle out of the gate. From one of the OP's comments, "Oracle support told him, step by step, to encrypt the database, which happened ages ago and is the root of all of this." That wallet is the responsibility of whomever created it. Just because it's empty now, doesn't mean it's always been empty. In addition to that, what if the person who initially encrypted everything with TDE, used a local only wallet? This would explain why the ewallet.p12 file is empty. Oracle recommends that you store your master key in the ewallet.p12 file and not the cwallet.sso file, which is the auto-open file, but only on the workstation it was created on. There could have been multiple failures here with multiple parties and not just Oracle.
hotsauceror 7y ago
I'd be very curious to know what drove that decision to recommend enabling encryption in the first place. That seems like an odd way to "solve a problem with backups", particularly given the huge risks inherent to irresponsible key management. It almost sounds like their original support engineer proposed this as a fix during a ticket. Normally something like this is something you'd plan out, test, and do risk analysis with the business rather than have your on-call deploy it in the middle of the night because John Q. Engineer said you should.

icedchai 7y ago

Sounds like they are using a managed database service. If that is the case, there is no system to administer and Oracle is at fault. Welcome to the cloud.

vbezhenar 7y ago

Yep. Probably Oracle has standard no warranties and obligations clause in their license.

djsumdog 7y ago

This is Oracle's hosted solution (like Amazon RDS) and their software failed. The OP even mentioned how the backups of their wallet were empty and this was never flagged somehow. If this was something they ran locally, yet it'd probably be their fault. But this is a hosted solution with a major bug. I hope some Oracle people are searching through seeing how many other customers have this empty wallet bug, and fixing it now.

justinclift 7y ago
Thinking it through, if they find other wallets with this problem, I wonder what the fix is? They could generate a new key + wallet, which could probably then be made to work for the running instance. All the backups though, would probably need to be opened with the current (missing) key+wallet in order to re-encrypt them. Might be doable with the current key+wallet that's in memory, but also it might not be. eg might not be able to change the software on the running system, in which case ouch... places that need historical backups (eg for legal purposes) would be in bad place.