4 ms·

So what's the current optimal solution, as far as precautionary measurements go - for these kinds of scenarios? The more companies that shell out, the more it'

So what's the current optimal solution, as far as precautionary measurements go - for these kinds of scenarios? The more companies that shell out, the more it's going to happen / motivate these pirates to continue with such rackets.

arkadiyt 6y ago
Take backups, practice restoring them, invest in a security program.
- stu2b50 6y ago
  Just having backups isn't enough. It's ransomware 101 to wait and try to encrypt all of their online backups and any recent offline backups. You really need comprehensive, air-gapped backups that date to years back.
  - drdaeman 6y ago
    > It's ransomware 101 to wait Not really. A sister company of my previous company had ransomware incident, and as far as I've heard this was not the case. They had just purged local backups. The attack was stopped quite early by an engineer noticing abnormally high IO activity and shutting the whole infrastructure down as soon as they realized what's going on - while a lot of data was lost and had to be restored from backups, most of the files were untouched. Still, the recovery took really long while to audit every machine before they could be even powered on again. Poisoning backups requires backup systems receiving encrypted data for a while. Which means live systems running off the encrypted data (and most ransomware encrypts at the file level, which is much harder to do transparently, compared to the block device level). Which requires effort to make sure this is extremely transparent and goes unnoticed. Doubt that attackers do expend their resources unless they see a necessity.
  - john_moscow 6y ago
    That's why you want incremental off-site backups that trigger alerts when too many files change at once.
drdaeman 6y ago
Continuous append-only backups, where one can't rewrite them without physical access to the system, would - most likely - help with a data loss, malicious or accidental.
- paulgb 6y ago
  Maybe a dumb question but -- I'm not a sysadmin, why isn't this the case already? At least for a company doing $1.5 billion in revenue.
  - drdaeman 6y ago
    Same as with purchasing [non-mandatory] insurance, I guess. One may not recognize the real value of something (or costs of losing something) until it happens.
zanny 6y ago
Offsite offline backups. Redundant ones. This scenario is really no different than if your datacenter had a gas leak and blew up, particularly because you never will ever be able to prove the attacker didn't retain your data somewhere so all you can do is guarantee recovery. Of note in this case is that the thieves only stole 2TB of data - this is a trivial expense to orchestrate a manyfold backup regime for in near real time when the alternative is shelling out millions. And its also about your threat model. If data leakage of any form threatens your business you need way more security than if you just want to be able to recover from exploits in your publicly facing infrastructure (or the ability for a rogue actor inside the company from sabotaging the business from the inside). At the most extreme having physical separation of infrastructure with physical token based auth and multiple signature verification to interact with data is going to be a heavy price in diligence to maintain secrecy. At the lowest end having a redundant backup storage array with a cron job on all employee computers to versioned backup files every minute that doesn't have network signin access.
- TwoBit 6y ago
  As long as the backed up data can't be infected.
the8472 6y ago
The responses suggesting backsups are ignoring the exfiltration part. The ransomware groups have updated their strategy to encrypt and carry away data that they would leak if not paid. Protecting against that is much harder. Compartmentalization and data minimization might help.
- rectang 6y ago
  "Data is a toxic asset." If it would hurt the business when leaked, then its value has to justify the risk of keeping it around.
Veserv 6y ago
Do you want a solution that makes such an attack unprofitable to execute (i.e. it raises the cost of doing a similar attack above $4.5M)? Buy insurance and pray. There is not a single readily available enterprise solution that would even dare to put that in writing, let alone deliver. If there is one that does dare, ask to test their claim by having the deal conditional on them putting out an open $4.5M bug bounty and nobody collecting on it (you should also be the one to determine if they have to pay). Every company will either back out or get collected on, no question. This is a good test because if it costs more than $4.5M to do such an attack, then it would be unprofitable to collect on the bug bounty. It is also unlikely to give a false positive (their system is good when it is not) since $4.5M is more than than almost every other bug bounty and it is totally above board, so you will get the best of the best trying to break into the system.
- jfoster 6y ago
  Not a good test. If they expect a decent ROI, the attack can't cost any amount up to $4.5M to execute. The $4.5M isn't a guaranteed outcome from such an attack. Some targets may not be willing to pay, or may only be willing to pay a substantially smaller amount. Also, part of the execution cost is the non-monetary "breaking the law" factor. For instance, would you rather make a legal $1M or an illegal $2M?
  - Veserv 6y ago
    Are you saying it is a bad test because it overestimates the difficulty? If so, I agree that it overestimates the difficulty. My argument is that nobody would dare to accept such a test, which is clearly an overestimate of difficulty, therefore nobody is even close to achieving the actual number. If you wanted a more accurate estimate of security you would probably need to bump it up by a factor of 3-5x to account for execution risk. If you are not, I am not sure what you are arguing since all of your statements show how the test overestimates the difficulty. The test is designed to identify, with reasonably high confidence, whether any attack is profitable given a specific upside. Therefore it should be as easy as possible for a legitimate bug that could result in at least a $4.5M upside to be paid. A 100% guaranteed legal payout clearly minimizes the risk and thus allows more attacks below a $4.5M cost to execute to be profitable. Any other form of payout means the cost to execute must be lower to be profitable. Put another way, if it is unprofitable to do it totally legally for some amount of money, it is probably even more unprofitable to do it illegally for the same amount of money (obviously this excludes cases where you might be able to gain a higher upside, but then we are not talking about mitigating attacks with a certain upside), therefore this estimate should be no lower than the true cost to execute (on average). To use your examples: If they want a decent ROI, the cost of attack must be significantly less than $4.5M to execute to be a good investment. ROI of illegal actions usually needs to be higher to make up for the risk since you would almost always choose a legal action with the same ROI. If the probability of payout is less than 100%, then the cost of attack must be less than $4.5M to make up for the reduced probability of success. The probability of payout for a bug bounty is usually higher than the highly variable payout of an attack. Also since the buyer is using this to make a quality decision and has authority to force the vendor to pay out with the stated scheme, it is in the buyers best interest to pay out credible attacks. If the non-legal nature is a serious cost, then the cost of attack must be less than $4.5M for the illegal case to make up for the extra risk and cost. Non-legal actions usually come with extra costs compared to legal actions and thus have to be even more profitable to be worth doing. Therefore, if a $4.5M bug bounty (where payout is decided by the product buyer) is not claimed after some reasonable amount of time we can conclude, with some reasonable amount of confidence, that the lowest risk option is likely unprofitable. Therefore, higher risk illegal options with the same upside are even less likely to be profitable. Thus, the test is a relatively good lower-bound for identifying if attacks with a $4.5M upside are actually being mitigated. If you can not even institute this lower bound, then you are nowhere near the necessary level.

arkadiyt 6y ago

Take backups, practice restoring them, invest in a security program.

stu2b50 6y ago
Just having backups isn't enough. It's ransomware 101 to wait and try to encrypt all of their online backups and any recent offline backups. You really need comprehensive, air-gapped backups that date to years back.
- drdaeman 6y ago
  > It's ransomware 101 to wait Not really. A sister company of my previous company had ransomware incident, and as far as I've heard this was not the case. They had just purged local backups. The attack was stopped quite early by an engineer noticing abnormally high IO activity and shutting the whole infrastructure down as soon as they realized what's going on - while a lot of data was lost and had to be restored from backups, most of the files were untouched. Still, the recovery took really long while to audit every machine before they could be even powered on again. Poisoning backups requires backup systems receiving encrypted data for a while. Which means live systems running off the encrypted data (and most ransomware encrypts at the file level, which is much harder to do transparently, compared to the block device level). Which requires effort to make sure this is extremely transparent and goes unnoticed. Doubt that attackers do expend their resources unless they see a necessity.
- john_moscow 6y ago
  That's why you want incremental off-site backups that trigger alerts when too many files change at once.

drdaeman 6y ago

Continuous append-only backups, where one can't rewrite them without physical access to the system, would - most likely - help with a data loss, malicious or accidental.

paulgb 6y ago
Maybe a dumb question but -- I'm not a sysadmin, why isn't this the case already? At least for a company doing $1.5 billion in revenue.
- drdaeman 6y ago
  Same as with purchasing [non-mandatory] insurance, I guess. One may not recognize the real value of something (or costs of losing something) until it happens.

zanny 6y ago

Offsite offline backups. Redundant ones. This scenario is really no different than if your datacenter had a gas leak and blew up, particularly because you never will ever be able to prove the attacker didn't retain your data somewhere so all you can do is guarantee recovery. Of note in this case is that the thieves only stole 2TB of data - this is a trivial expense to orchestrate a manyfold backup regime for in near real time when the alternative is shelling out millions. And its also about your threat model. If data leakage of any form threatens your business you need way more security than if you just want to be able to recover from exploits in your publicly facing infrastructure (or the ability for a rogue actor inside the company from sabotaging the business from the inside). At the most extreme having physical separation of infrastructure with physical token based auth and multiple signature verification to interact with data is going to be a heavy price in diligence to maintain secrecy. At the lowest end having a redundant backup storage array with a cron job on all employee computers to versioned backup files every minute that doesn't have network signin access.

TwoBit 6y ago
As long as the backed up data can't be infected.

the8472 6y ago

The responses suggesting backsups are ignoring the exfiltration part. The ransomware groups have updated their strategy to encrypt and carry away data that they would leak if not paid. Protecting against that is much harder. Compartmentalization and data minimization might help.

rectang 6y ago
"Data is a toxic asset." If it would hurt the business when leaked, then its value has to justify the risk of keeping it around.

Veserv 6y ago

Do you want a solution that makes such an attack unprofitable to execute (i.e. it raises the cost of doing a similar attack above $4.5M)? Buy insurance and pray. There is not a single readily available enterprise solution that would even dare to put that in writing, let alone deliver. If there is one that does dare, ask to test their claim by having the deal conditional on them putting out an open $4.5M bug bounty and nobody collecting on it (you should also be the one to determine if they have to pay). Every company will either back out or get collected on, no question. This is a good test because if it costs more than $4.5M to do such an attack, then it would be unprofitable to collect on the bug bounty. It is also unlikely to give a false positive (their system is good when it is not) since $4.5M is more than than almost every other bug bounty and it is totally above board, so you will get the best of the best trying to break into the system.

jfoster 6y ago
Not a good test. If they expect a decent ROI, the attack can't cost any amount up to $4.5M to execute. The $4.5M isn't a guaranteed outcome from such an attack. Some targets may not be willing to pay, or may only be willing to pay a substantially smaller amount. Also, part of the execution cost is the non-monetary "breaking the law" factor. For instance, would you rather make a legal $1M or an illegal $2M?
- Veserv 6y ago
  Are you saying it is a bad test because it overestimates the difficulty? If so, I agree that it overestimates the difficulty. My argument is that nobody would dare to accept such a test, which is clearly an overestimate of difficulty, therefore nobody is even close to achieving the actual number. If you wanted a more accurate estimate of security you would probably need to bump it up by a factor of 3-5x to account for execution risk. If you are not, I am not sure what you are arguing since all of your statements show how the test overestimates the difficulty. The test is designed to identify, with reasonably high confidence, whether any attack is profitable given a specific upside. Therefore it should be as easy as possible for a legitimate bug that could result in at least a $4.5M upside to be paid. A 100% guaranteed legal payout clearly minimizes the risk and thus allows more attacks below a $4.5M cost to execute to be profitable. Any other form of payout means the cost to execute must be lower to be profitable. Put another way, if it is unprofitable to do it totally legally for some amount of money, it is probably even more unprofitable to do it illegally for the same amount of money (obviously this excludes cases where you might be able to gain a higher upside, but then we are not talking about mitigating attacks with a certain upside), therefore this estimate should be no lower than the true cost to execute (on average). To use your examples: If they want a decent ROI, the cost of attack must be significantly less than $4.5M to execute to be a good investment. ROI of illegal actions usually needs to be higher to make up for the risk since you would almost always choose a legal action with the same ROI. If the probability of payout is less than 100%, then the cost of attack must be less than $4.5M to make up for the reduced probability of success. The probability of payout for a bug bounty is usually higher than the highly variable payout of an attack. Also since the buyer is using this to make a quality decision and has authority to force the vendor to pay out with the stated scheme, it is in the buyers best interest to pay out credible attacks. If the non-legal nature is a serious cost, then the cost of attack must be less than $4.5M for the illegal case to make up for the extra risk and cost. Non-legal actions usually come with extra costs compared to legal actions and thus have to be even more profitable to be worth doing. Therefore, if a $4.5M bug bounty (where payout is decided by the product buyer) is not claimed after some reasonable amount of time we can conclude, with some reasonable amount of confidence, that the lowest risk option is likely unprofitable. Therefore, higher risk illegal options with the same upside are even less likely to be profitable. Thus, the test is a relatively good lower-bound for identifying if attacks with a $4.5M upside are actually being mitigated. If you can not even institute this lower bound, then you are nowhere near the necessary level.