6 ms·

It should be noted that Colonial had several infosec openings at the time of the attack. While having those filled might not have prevented this attack, it also

It should be noted that Colonial had several infosec openings at the time of the attack. While having those filled might not have prevented this attack, it also might have or at least put them in a better response position. There are lots of infosec openings across the country but compensation doesn't seem to be rising in response. It appears that companies are fine with leaving these positions open for long periods of time. As long as the position actually exists, they're not all that concerned with filling it. This might be complacency creep. Everyone staffed up after the cluster of breaches that happened around the time of the Target and Equifax breaches. A lack of other high profile breaches or attacks might be why many companies have become lax in keeping their staffs full.

medicineman 5y ago
I mean, if you ignore how H1B's work, yeah you could call it complacency.
sneak 5y ago
You don't need infosec staff to know that you should have backups of the data on your important computers/servers. Being hit by ransomware is not an indicator of total IT incompetence. Having no good options but to pay the ransom absolutely is. All ransomware is doing is exposing the existing hope-based DR plans (that is to say, lack thereof) in the industry.
- alistairSH 5y ago
  Being hit by ransomware is not an indicator of total IT incompetence. Having no good options but to pay the ransom absolutely is. Part of paying the ransom is the promise that the ransomer will not just unlock your system, but will also delete all the data they downloaded (which often includes a pile of PII that the ransomee doesn't want published).
- elliekelly 5y ago
  I wouldn’t even pin it on IT. I’d be willing to bet there’s some poor IT person, or perhaps a lot of them, cleaning up this mess who have been begging management to beef up info sec for a long time.
- jiggawatts 5y ago
  I used to deploy backup systems, and I had to explain to many customers that data protection is a security feature also. Very few "got" this.
- rapjr9 5y ago
  Note that the attackers can also threaten to release data. Backups are no protection against that. They could also corrupt data and not tell you which part was corrupted and when, so even if you have backups you don't know which ones are corrupted unless you have some way of verifying all the data. One example of this would be to plant a backdoor, leave it in place and unused for months, then trigger the ransomeware encryption. The company decides not to pay, they restore from backup, and the attackers use the backdoor to encrypt it all again and demand even more. They could also use access to destroy hardware, say on a timer that triggers after the payment deadline. Backups won't protect against that if you can't get all your systems offline fast enough, or if taking them offline triggers the destruction.
MattGaiser 5y ago
Are there enough Infosec people to fill every open job for it in the USA? I would imagine that it is like software development, where the unemployed software devs are the kind that can't figure out git.
- lawnchair_larry 5y ago
  This is basically accurate but with an added problem. When devs do their job, the product is software. When security does their job, the product is “not getting hacked”, so if you act busy enough, it’s easy to appear as though you’re doing important work, until it’s too late. Then, paradoxically, you aren’t actually punished, but usually rewarded, when you do get hacked. That’s the one time you’re needed most, and you get to act like the hero for saving the company.
- Mountain_Skies 5y ago
  I doubt there are enough infosec people which means in theory that compensation should rise which will then attract more people into the field. Until they're trained and experienced, whoever provides the best place to work (compensation and intangibles that lead to satisfaction) would get the help they need while others would be more vulnerable to attack. But from what I've seen, this isn't happening. There's lots of complaints about there not being enough workers but instead of boosting compensation and/or quality of employment, the positions simply stay open for extended periods of time.
  - faeyanpiraat 5y ago
    What would be the solution to this? If I’m a leader in a company with a culture and intangibles not yet optimized for the people working in the infosec roles, how would I aproach changing the environment for the better? Is it viable to cooperate with other companies to share best practices? Wouldn’t they hesitate to share? Would doing deep interviews with potential employees get me the right information? Would some hr consultancy provide this info? Arent’t they too old-fashioned for this field yet?
orev 5y ago
For many companies, security threats are all theoretical, but they are required to have the positions to meet some compliance requirement. They need to have them, but don’t really want them, which would explain the lack of enthusiasm (as demonstrated by the low salaries) in getting the jobs actually filled. Also, a lot of infosec positions are just chugging through audits and ticking boxes to say whether you have some control in place or not. Those are more clerical positions that don’t require deep technical knowledge that could command a higher salary.
lawnchair_larry 5y ago
The issue is less about people unwilling to take those wages, and more about a lack of people whose breath can even fog a security mirror so to speak. I work in security and have been involved with hiring at several “brand name” companies including FAANGs in hot tech markets, and it’s always been a talent pipeline issue more than anything. Given how difficult it is for the biggest players to keep security staffed up, and they still get hacked routinely, I can’t imagine how low quality the applicant pool is at Colonial, and doubt it would have made a difference. Almost every company of moderate size perpetually has openings for security roles. The other problem is that the industry has an oversupply of by-the-book certified security people who can configure firewalls and run scanners, but who have never dealt with live hackers or hacked anything themselves. But hackers are clever and artistic, and defending against them isn’t like following a recipe for baking a cake. And as an employer looking to introduce security, there is no way to really evaluate a good security leader vs a charlatan, and then it’s either bad hires all the way down, or talented people on the bottom who lack leadership and are ineffective in the bureaucracy.
- grumple 5y ago
  The problem I see is that there are tradeoffs between security and usability, and again between developing security vs developing features. Security doesn't make money next quarter, while features and ease of use do. Any software engineer can do security if they spend time learning and working on it. But executives don't seem to care about it.
  - lawnchair_larry 5y ago
    Weird to see this downvoted, it’s completely accurate and pretty basic economics. Security is a cost center, product development is a revenue multiplier. Investing in the latter as much as you can get away with is the most rational way to allocate resources.
- socialist_coder 5y ago
  Is being a "good" security person really more involved than: * making sure you have all your ports locked down * limit connectivity between all instances to only the bare minimum * any public access is via protocols such as ssh which have zero-to-none vulnerabilities * any 3rd party software you dont know is secure should never be public * routinely run employee training on how not to let themselves get hacked via social engineering I'm sure I'm missing other stuff, but I feel like if you follow these "best practices", you have just made yourself a very hard target and hackers will probably skip over you unless they have some weird reason to target your org specifically. So for 95% of companies out there, this level of security should be sufficient. I'm legitimately asking - is this sufficient? Or are hackers so creative that even following these basic rules will still not make you a hard target? This stuff seems fairly easy to do but I agree you need training or an info-sec person making sure your dev teams are doing it all. You can't have any slip ups. Your devs / managers have to take it seriously.
hn8788 5y ago
From my experience, the problem is that most infosec positions are powerless to do anything to increase security at the company, and are primarily there for PR or compliance reasons. The positions seem to be mostly filled with people who wanted to make a career change for the money; experienced people usually leave to work at private security companies, or FAANG sized companies.
- ianhawes 5y ago
  This. A million times this. I can’t tell you how many netsec roles are staffed by people that are content being a butt in a seat and have zero effect on the overall security of a corporation.
coldpie 5y ago
I mean, let's address the elephant in the room: there is no such thing as computer security. As we see with new leaks and hacks and vulnerabilities every single week, the idea that a computer that is connected to the Internet can be secure is a joke. The whole industry is built on protocols and tools that assume there will never be any bad actors, and we're reaping the rewards of that now. It will take decades of layering on band-aids to approach anything like security, and more likely we will have to rebuild the entire industry from the ground up without that assumption. Both will take a very long time and a lot of money. Hiring some guy with an infosec cert would not have stopped this attack, because there is no way to stop this kind of attack.
- TameAntelope 5y ago
  There are companies that get hacked a lot and there are companies that don't. It is for sure true to say everyone is vulnerable, but it's also true to say that you can reduce your risk without reducing your revenue.
- Aperocky 5y ago
  > Idea that a computer that is connected to the Internet can be secure is a joke. The whole industry is built on protocols and tools that assume there will never be any bad actors This is just flat out wrong.
- Mountain_Skies 5y ago
  Risk cannot be eliminated but it certainly can be reduced. Also response plans for when something happens can be funded and regularly tested. You can't anticipate every possible successful attack but you can reduce the risk of being unprepared to respond to whatever attack happens.
- mistrial9 5y ago
  the assumption that there is no security in open protocols is badly misinformed here. "Hiring some guy with an infosec cert would not have stopped this attack, because there is no way to stop this kind of attack." blovation
Veserv 5y ago
No. The security problem is not a lack of effort or laxness, it is a fundamental inability to solve the problem. At a $5M payout there are essentially 0 commercial IT systems in the world that can stop such an attack. The absolute best of the best commercial IT systems implemented as envisioned with full support can maybe protect up to the $10M level and I am just extrapolating upwards since I have never had any security professional or executive in a Fortune 500 company with a budget in the tens to hundreds of millions of dollars ever assess their own systems as more than $1M. With an ROI of 5 is it only a matter of time before criminal enterprises can bootstrap themselves up to exploit the entire total addressable market. At best, better, but still inadequate, security means that the thousands of hungry bears eat the slower fish in the barrel first to get the energy to reproduce and make more bears to eat the rest. This is not a failure to live up to potential or incompetence, though there is a fair amount of both of those. We need solutions that are literally 100x better than the best systems currently available before we get to even adequate for critical infrastructure whose disruption can literally cause hundreds of millions or billions of dollars in damage let alone potential human lives. Anything less than that keeps extortion economically viable for the attackers and paying off extortion economically sound for the victims. That is how far away we are.
- abraae 5y ago
  > At a $5M payout there are essentially 0 commercial IT systems in the world that can stop such an attack. Even if that's true, it doesn't affect backups. Back your fucking systems up properly, and if you are attacked by ransomware, then do a scorched earth restore.
  - Veserv 5y ago
    It absolutely does affect backups. If you stand to gain $5M from an attack you can also target the backup systems and still easily end up profitable. Only if you stand to gain less than $100k does the budget actually start to get tight. As for how you attack the backup system it depends. If it push based you send your payload during the push. If it is pull based you craft your payload in the data that will be backed up. If it is not append-only you can easily nuke the entire available history. If it is append-only, but that is only done in software you just need to take over the software. If it is in hardware you just infiltrate then silently encrypt any new data until it would be painful to revert that far back in time. Given that the mean-time to discovery is on the order of months that is quite painful. If they regularly test their backups you just silently decrypt the data on restore until it is time to strike. There are plenty of ways to beat vulnerable backup systems in that sort of budget. Like, seriously, with a $5M budget you can literally purchase and burn multiple zero days for every system in the chain and still come out ahead. You can hire 10-50 full time software engineers for a year per attack. Most systems have serious vulnerabilities discovered by lone individuals working for a few months in their free time let alone a team of 50 people. The current backup systems survive because most of these attacks are being done with budgets closer to $10k-$100k to maximize profit and growth rate and that is not really enough money to pay for the second arm of the attack. But with a $5M return they could easily allocate a few million to capitalize on the opportunity if that is what is needed once all the juicier targets have been eaten.

medicineman 5y ago

I mean, if you ignore how H1B's work, yeah you could call it complacency.

sneak 5y ago

You don't need infosec staff to know that you should have backups of the data on your important computers/servers. Being hit by ransomware is not an indicator of total IT incompetence. Having no good options but to pay the ransom absolutely is. All ransomware is doing is exposing the existing hope-based DR plans (that is to say, lack thereof) in the industry.

alistairSH 5y ago
Being hit by ransomware is not an indicator of total IT incompetence. Having no good options but to pay the ransom absolutely is. Part of paying the ransom is the promise that the ransomer will not just unlock your system, but will also delete all the data they downloaded (which often includes a pile of PII that the ransomee doesn't want published).
elliekelly 5y ago
I wouldn’t even pin it on IT. I’d be willing to bet there’s some poor IT person, or perhaps a lot of them, cleaning up this mess who have been begging management to beef up info sec for a long time.
jiggawatts 5y ago
I used to deploy backup systems, and I had to explain to many customers that data protection is a security feature also. Very few "got" this.
rapjr9 5y ago
Note that the attackers can also threaten to release data. Backups are no protection against that. They could also corrupt data and not tell you which part was corrupted and when, so even if you have backups you don't know which ones are corrupted unless you have some way of verifying all the data. One example of this would be to plant a backdoor, leave it in place and unused for months, then trigger the ransomeware encryption. The company decides not to pay, they restore from backup, and the attackers use the backdoor to encrypt it all again and demand even more. They could also use access to destroy hardware, say on a timer that triggers after the payment deadline. Backups won't protect against that if you can't get all your systems offline fast enough, or if taking them offline triggers the destruction.

MattGaiser 5y ago

Are there enough Infosec people to fill every open job for it in the USA? I would imagine that it is like software development, where the unemployed software devs are the kind that can't figure out git.

lawnchair_larry 5y ago
This is basically accurate but with an added problem. When devs do their job, the product is software. When security does their job, the product is “not getting hacked”, so if you act busy enough, it’s easy to appear as though you’re doing important work, until it’s too late. Then, paradoxically, you aren’t actually punished, but usually rewarded, when you do get hacked. That’s the one time you’re needed most, and you get to act like the hero for saving the company.
Mountain_Skies 5y ago
I doubt there are enough infosec people which means in theory that compensation should rise which will then attract more people into the field. Until they're trained and experienced, whoever provides the best place to work (compensation and intangibles that lead to satisfaction) would get the help they need while others would be more vulnerable to attack. But from what I've seen, this isn't happening. There's lots of complaints about there not being enough workers but instead of boosting compensation and/or quality of employment, the positions simply stay open for extended periods of time.
- faeyanpiraat 5y ago
  What would be the solution to this? If I’m a leader in a company with a culture and intangibles not yet optimized for the people working in the infosec roles, how would I aproach changing the environment for the better? Is it viable to cooperate with other companies to share best practices? Wouldn’t they hesitate to share? Would doing deep interviews with potential employees get me the right information? Would some hr consultancy provide this info? Arent’t they too old-fashioned for this field yet?

orev 5y ago

For many companies, security threats are all theoretical, but they are required to have the positions to meet some compliance requirement. They need to have them, but don’t really want them, which would explain the lack of enthusiasm (as demonstrated by the low salaries) in getting the jobs actually filled. Also, a lot of infosec positions are just chugging through audits and ticking boxes to say whether you have some control in place or not. Those are more clerical positions that don’t require deep technical knowledge that could command a higher salary.

lawnchair_larry 5y ago

The issue is less about people unwilling to take those wages, and more about a lack of people whose breath can even fog a security mirror so to speak. I work in security and have been involved with hiring at several “brand name” companies including FAANGs in hot tech markets, and it’s always been a talent pipeline issue more than anything. Given how difficult it is for the biggest players to keep security staffed up, and they still get hacked routinely, I can’t imagine how low quality the applicant pool is at Colonial, and doubt it would have made a difference. Almost every company of moderate size perpetually has openings for security roles. The other problem is that the industry has an oversupply of by-the-book certified security people who can configure firewalls and run scanners, but who have never dealt with live hackers or hacked anything themselves. But hackers are clever and artistic, and defending against them isn’t like following a recipe for baking a cake. And as an employer looking to introduce security, there is no way to really evaluate a good security leader vs a charlatan, and then it’s either bad hires all the way down, or talented people on the bottom who lack leadership and are ineffective in the bureaucracy.

grumple 5y ago
The problem I see is that there are tradeoffs between security and usability, and again between developing security vs developing features. Security doesn't make money next quarter, while features and ease of use do. Any software engineer can do security if they spend time learning and working on it. But executives don't seem to care about it.
- lawnchair_larry 5y ago
  Weird to see this downvoted, it’s completely accurate and pretty basic economics. Security is a cost center, product development is a revenue multiplier. Investing in the latter as much as you can get away with is the most rational way to allocate resources.
socialist_coder 5y ago
Is being a "good" security person really more involved than: * making sure you have all your ports locked down * limit connectivity between all instances to only the bare minimum * any public access is via protocols such as ssh which have zero-to-none vulnerabilities * any 3rd party software you dont know is secure should never be public * routinely run employee training on how not to let themselves get hacked via social engineering I'm sure I'm missing other stuff, but I feel like if you follow these "best practices", you have just made yourself a very hard target and hackers will probably skip over you unless they have some weird reason to target your org specifically. So for 95% of companies out there, this level of security should be sufficient. I'm legitimately asking - is this sufficient? Or are hackers so creative that even following these basic rules will still not make you a hard target? This stuff seems fairly easy to do but I agree you need training or an info-sec person making sure your dev teams are doing it all. You can't have any slip ups. Your devs / managers have to take it seriously.

hn8788 5y ago

From my experience, the problem is that most infosec positions are powerless to do anything to increase security at the company, and are primarily there for PR or compliance reasons. The positions seem to be mostly filled with people who wanted to make a career change for the money; experienced people usually leave to work at private security companies, or FAANG sized companies.

ianhawes 5y ago
This. A million times this. I can’t tell you how many netsec roles are staffed by people that are content being a butt in a seat and have zero effect on the overall security of a corporation.

coldpie 5y ago

I mean, let's address the elephant in the room: there is no such thing as computer security. As we see with new leaks and hacks and vulnerabilities every single week, the idea that a computer that is connected to the Internet can be secure is a joke. The whole industry is built on protocols and tools that assume there will never be any bad actors, and we're reaping the rewards of that now. It will take decades of layering on band-aids to approach anything like security, and more likely we will have to rebuild the entire industry from the ground up without that assumption. Both will take a very long time and a lot of money. Hiring some guy with an infosec cert would not have stopped this attack, because there is no way to stop this kind of attack.

TameAntelope 5y ago
There are companies that get hacked a lot and there are companies that don't. It is for sure true to say everyone is vulnerable, but it's also true to say that you can reduce your risk without reducing your revenue.
Aperocky 5y ago
> Idea that a computer that is connected to the Internet can be secure is a joke. The whole industry is built on protocols and tools that assume there will never be any bad actors This is just flat out wrong.
Mountain_Skies 5y ago
Risk cannot be eliminated but it certainly can be reduced. Also response plans for when something happens can be funded and regularly tested. You can't anticipate every possible successful attack but you can reduce the risk of being unprepared to respond to whatever attack happens.
mistrial9 5y ago
the assumption that there is no security in open protocols is badly misinformed here. "Hiring some guy with an infosec cert would not have stopped this attack, because there is no way to stop this kind of attack." blovation

Veserv 5y ago

No. The security problem is not a lack of effort or laxness, it is a fundamental inability to solve the problem. At a $5M payout there are essentially 0 commercial IT systems in the world that can stop such an attack. The absolute best of the best commercial IT systems implemented as envisioned with full support can maybe protect up to the $10M level and I am just extrapolating upwards since I have never had any security professional or executive in a Fortune 500 company with a budget in the tens to hundreds of millions of dollars ever assess their own systems as more than $1M. With an ROI of 5 is it only a matter of time before criminal enterprises can bootstrap themselves up to exploit the entire total addressable market. At best, better, but still inadequate, security means that the thousands of hungry bears eat the slower fish in the barrel first to get the energy to reproduce and make more bears to eat the rest. This is not a failure to live up to potential or incompetence, though there is a fair amount of both of those. We need solutions that are literally 100x better than the best systems currently available before we get to even adequate for critical infrastructure whose disruption can literally cause hundreds of millions or billions of dollars in damage let alone potential human lives. Anything less than that keeps extortion economically viable for the attackers and paying off extortion economically sound for the victims. That is how far away we are.

abraae 5y ago
> At a $5M payout there are essentially 0 commercial IT systems in the world that can stop such an attack. Even if that's true, it doesn't affect backups. Back your fucking systems up properly, and if you are attacked by ransomware, then do a scorched earth restore.
- Veserv 5y ago
  It absolutely does affect backups. If you stand to gain $5M from an attack you can also target the backup systems and still easily end up profitable. Only if you stand to gain less than $100k does the budget actually start to get tight. As for how you attack the backup system it depends. If it push based you send your payload during the push. If it is pull based you craft your payload in the data that will be backed up. If it is not append-only you can easily nuke the entire available history. If it is append-only, but that is only done in software you just need to take over the software. If it is in hardware you just infiltrate then silently encrypt any new data until it would be painful to revert that far back in time. Given that the mean-time to discovery is on the order of months that is quite painful. If they regularly test their backups you just silently decrypt the data on restore until it is time to strike. There are plenty of ways to beat vulnerable backup systems in that sort of budget. Like, seriously, with a $5M budget you can literally purchase and burn multiple zero days for every system in the chain and still come out ahead. You can hire 10-50 full time software engineers for a year per attack. Most systems have serious vulnerabilities discovered by lone individuals working for a few months in their free time let alone a team of 50 people. The current backup systems survive because most of these attacks are being done with budgets closer to $10k-$100k to maximize profit and growth rate and that is not really enough money to pay for the second arm of the attack. But with a $5M return they could easily allocate a few million to capitalize on the opportunity if that is what is needed once all the juicier targets have been eaten.