15 ms·

Hackers breached Colonial Pipeline using compromised password

240 points by 1cvmask 5y ago

neonate 5y ago
https://archive.is/p2XY1 https://archive.is/p2XY1
SergeAx 5y ago
> The hack that took down the largest fuel pipeline in the U.S. This needs to be remembered: the pipeline was okay all the time. It is accounting software that was taken down. A conspiracy theorist in me thinks that most of the PR activity around the case was (and is) intending to paint this fact over.
upofadown 5y ago
The big issue here is that a single access could so easily be spun up into complete access. In a large company you have to assume that even the highest wall will eventually be breached. A current employee will eventually provide access to someone up to no good. Or that current employee will themselves get up to no good. Corporate messaging is a good example of this sort of thing. Completely unencrypted inside the wall. Tons of login information just left laying around in old plaintext messages. Your security should not be all or nothing...
vlan0 5y ago
When will people realize we need to stop implementing security in this manner? The use of passwords and perimeter security should be DEAD. https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-171r2.pdf https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.S... https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-207.pdf https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.S...
lbriner 5y ago
As others have said, MFA would be a decent tool but not all attacks are against VPN so it wouldn't always be so useful. As everyone always teaches, security has to be in-depth and the one largely simple control that is very often missing is segmentation of the system. Being able to attack a single department is annoying but unlikely to be considered fatal, however, I think that even savvy IT ops people do not really understand vlans, using switches and air-gaps to keep things apart and really concentrating on any single-points of ingress like Sharepoint sites or whatever although most problems seem to spread across the network layer rather than bridging machines over HTTP. Also, even some simple controls like basic local backup, even to a crappy SAN box in your server room would at least enable you to stick your fingers up at ransomware.
lumost 5y ago
At this point I either use saved passwords or need to rotate the passwords on every request via password reset. Saved passwords are also an attack vector and as such my corporate laptop disables all mechanisms for saving passwords. I think it's time to make pin + 2 factor the standard over passwords.
rdiddly 5y ago
Compromised password of an inactive account. Do regular audits and deactivate unused accounts, people! At the very least that takes care of the "disgruntled former employee" scenario. There's a lot of virtual ink in this article devoted to the "how were the credentials compromised" question. I understand the curiosity, but it's mostly unknowable and therefore not too interesting. It's like asking "Why did it rain on the day I didn't bring my umbrella?" Is that the question? "Why did it rain?" The interesting part is the point at which Colonial still had control and gave it up, which was when that employee left or whatever happened, and they didn't revoke those credentials. I keep having a thought about how this account belonged to a fired employee who was not only bad at passwords, but bad at this job too. I have no idea if that's the case, as that's generally confidential info. Kinda funny to think of though.
nabla9 5y ago
In this and age, if companies are not giving employees FIDO2 security keys it's just incompetence.
- arkadiyt 5y ago
  The account didn't have any kind of 2fa - even SMS would have stopped this breach. Obviously hardware keys are the gold standard but I wouldn't trivialize the amount of work it takes to roll that out to an organization.
  - Godel_unicode 5y ago
    I would, it's pretty trivial (I have done it for multiple organizations of different sizes/technical competencies). Especially for something like a VPN; at this point if you use a VPN solution which doesn't support MFA it almost certainly has other security issues as well.
  - ineedasername 5y ago
    I have a family member who had a very junior role at a big finance company 15 years ago and they used hardware keys. Seems like national infrastructure should require similar levels of security.
  - qbasic_forever 5y ago
    For any sysadmins or future sysadmins reading this though, do NOT put your service's 2FA on SMS. Do not use SMS for anything other than sending your grandparents a happy birthday message. It is trivially broken with a sim card hijack and we have seen many high profile incidents of people being hacked precisely through this vector.
    - rossdavidh 5y ago
      ...and yet, it would have been better than this setup.
  - kevin_thibedeau 5y ago
1cvmask 5y ago
They failed to implement multi-factor authentication on their VPNs. This is one of the most common attack vectors. VPNs by themselves are not enough. Over two thirds of corporate attacks with VPNs have been through compromised static credentials. There are numerous ways of integrating VPNs with MFA ranging from a RADIUS integration to a SAML one. At Saas Pass the admin can choose and implement up to 15 different methods of MFA to VPNs ranging from FIDO keys to push approval to scanning encrypted barcodes to SMS (not recommended) etc... Disclaimer: Worked on the design of some of the mfa interfaces for the SAML based VPNs and the access control policies.
- jl2718 5y ago
  Personally I think RADIUS is a terrible idea. If that thing is compromised, or even the network or name service to connect to it, the attacker gets everything. Cert-based auth gives every client the ability to independently authorize every login without any ability to learn or compromise credentials. And there are lots of cert-based MFAs that require nothing more than a few lines in an Apache config, or even handled through the operating system to set up a vpn handshake. One of my biggest contributions at one place I worked was to dismantle their radius auth and replace it with token cert auth. It’s hard to write about that on a resume, because people tend to be more impressed with complexity than simplicity.
  - er4hn 5y ago
    "Upgraded legacy, obsolete, AAA service with modern and standards compliant solution" And if it is ever asked about, boy do you have some great stories about upgrading from a system whose protocol name referred to dial in modems and used MD5 to obfuscate passwords.
    - qazxcvbnmlp 5y ago
      Thank you. Not the op but I have trouble coming up with this stuff for my own resume.
- g_p 5y ago
  One issue in the critical infrastructure space is that many of these MFA solutions are less robust and resilient than no MFA. Anything based on calling out to an external service (Duo, for example) is likely to be less resilient [1] than the static, no-external-dependencies system they use for plain password Auth. Clearly moving towards complex systems like RADIUS and SAML introduce even more moving parts that sit with interconnects between trusted and untrusted, internal and external components. That becomes a point of failure in the eyes of infrastructure companies. The challenge here seems to be getting the IT world to understand the focus on availability in critical infrastructure - historically, the focus has been on availability first, integrity second, confidentiality third. You could argue this shouldn't apply to VPN interfaces in to manage systems, but given the external dependencies of these systems, they feel far less likely to remain available than the kind of solutions the sector expects. On a separate note though, critical infrastructure need to re-think their budgets to pay for their increased availability needs, given most of the commercial solutions are third party hosted "as a service" offerings. [1] https://status.duo.com/history?page=2 https://status.duo.com/history?page=2
3pt14159 5y ago
Yes, MFA would have helped here, but I think it's besides the point. I've worked on a lot of projects, including one for a major stock exchange, and I've never seen perfect security. It's too slow. Even if you had MFA for the VPN, it doesn't stop local malware from being able to use the foothold to gain access to the network once the MFA is cleared. I'm not saying that MFA is useless, just that these types of things are very hard to get right and we only hear about the times the attackers got through. "Oh they should have done X" where X is any one of an increasingly long list of possible things they should have done ranging from security headers on their website, to MFA, to auditing their third party libraries, to locking down their development machines and networks (during Covid, ha!) it's never ending. The very best at it get by without total calamity, but the long tail of organizations just don't have the right staff, focus, or resources to do it. Edit: I'm not saying it's right! I don't think it is right, and I've worked with governments to bring in regulations in certain industries, but to characterize this all on one failure kinda misses the point. There is a broader, harder problem that I'm not sure is even solvable.
- genmud 5y ago
  Going to have to hard disagree there. MFA and addressing password reuse is table stakes in 2021. Like this is literally the first item I would do as a CISO in any company. MFA is among the core security items like running endpoint protection, updating your software or filtering spam. Bucketing it with things like CORS or auditing third party libraries is either disingenuous or extremely naive.
  - 3pt14159 5y ago
    Look, MFA for access to code repositories or a cloud is one thing, but if you think that MFA is "table stakes" for a VPN I don't know what to tell you other than "then very few orgs are doing table stakes with their VPNs." These types of arguments are so hard to have over HN, because people with actual experience across all sorts of organizations can't explicitly say what they have seen and where because of NDAs and we all have our own viewpoints and individual experience. Bucketing MFA into one broad concept is as disingenuous as calling out security headers as being as benign as CORS. Does your org have MFA on SSH? On email? On the code repo? On the Slack login? On the VPN(s)? Etc. Etc. I have yet to see a single org that had MFA on everything and blaming a lack of MFA on the VPN misses the point. If they'd had it there, the attackers would have hit elsewhere. Most developers install random software and browser toolbars that they'll never have time to check and connect directly to their companies servers, repos, and communication tools. Pretending like MFA on a VPN would have magically solved everything is a pipedream. There are too many ways in, and every time someone gets in the response from the software community is always "oh why didn't they do X" while ignoring the larger truth: There are too many threat vectors and getting them all right, at all times, is nigh on impossible.
dmitrygr 5y ago
I wonder if a useful counter-measure would be slower internet connectivity it such places. It would be very hard to exfiltrate 100GB at 8kbit speeds, while 8kbit is plenty for simple few-byte commands one would need to manage a pipeline.
- goodpoint 5y ago
  It is, and data diodes are even better.
- scoopertrooper 5y ago
  From what I understand, the hackers never got anywhere near the control systems. The systems they compromised were related to billing and probably the company's sharepoint and emails (hard to think what else the could make up the balance of the 100GB outside of the monitoring of the control systems). You could still slow down attacks like this by limiting bandwidth (though 8kbits might be painful for document sharing). However, if you limited users a _slightly_ more reasonable 1 megabit per second, they'd be able to complete their exfiltration in within a little over 9 days. Keep in mind, Colonial only learned of the attack after the ransom demand was made!
  - rapjr9 5y ago
    The article says the ransomware message appeared on a computer in the control room. That seems strange, do the pipeline operators adjust fuel flow based on how much cash is coming in? Is this a common practice? Or maybe it was not the billing system that was compromised? I guess it is possible billing would be used to predict required flow rate in some way, but that means any hack of the internet connected billing system has a direct route to controlling fuel flow in the pipeline, which seems rather dangerous. Post some fake orders and suddenly there are millions of gallons of fuel wherever you say they should be. Does the pipeline also run in reverse? If not it then that fuel could be stranded.
    - cesarb 5y ago
      > The article says the ransomware message appeared on a computer in the control room. That seems strange That computer was probably their "office" computer, which they would use for things like email. From what I understood from the article ("[...] an employee in Colonial’s control room saw a ransom note demanding cryptocurrency appear on a computer just before 5 a.m. The employee notified an operations supervisor who immediately began to start the process of shutting down the pipeline, [...]. By 6:10 a.m., the entire pipeline had been shut down"), the supervisor made the sensible decision of immediately starting an orderly shutdown of the pipeline, before the thing propagated to their control computers; as can be seen by the timing, doing an orderly shutdown takes a while.
vmception 5y ago
Password found in other leaks No multi factor authentication Wants government to go after people Government should go after Colonial
- ineedasername 5y ago
  Sure, penalties for negligent companies might help, but if someone leaves the front door open and gets robbed, you still go after the thieves too. Although the idea that the east coast pipeline infrastructure could be taken down by the rough equivalent of keeping the door open it pretty terrifying in its own right.
  - alexanderdmitri 5y ago
    The door was closed and locked, but they a) didn't revoke an old key that got tied to it and b) didn't attempt to verify it was expected person using it when it was tried on the door. Making the distinction because I don't believe they were explicitly inviting people to rob them as much as they didn't take sufficient precautions in a bad neighborhood.
    - gwbas1c 5y ago
      Honestly, we can't let basic security turn into an arms race. It's getting harder and harder to maintain good password practice. Part of good security also needs to involve proactively going after the criminals. Anyone willing to break a window can break into my home. Internet security needs to be the same way: Secure enough to keep the honest people honest, and government agencies need to track down the hackers and imprison them. We can't expect normal people to perfectly implement complicated security procedures all the time, nor can we have pervasive technology and only run it with the world's smartest and most perfect people.
      - ridaj 5y ago
        Of course there needs to be consequences for thieves but that doesn't absolve people (particularly in critical infrastructure) from the responsibility to implement reasonable protections. If everyone keeps their doors unlocked in town and hundreds of burglaries occur as a result, the police should do something about it, but they may not be staffed to go after everyone. At some point they're going to be telling people to lock their houses please. "Password practice" is thankfully becoming less and less relevant as hardware security tokens become more widespread, affordable and reliable. IMO there's zero excuse for a critical infrastructure operator not to use 2FA at this point. Whatever the password policy is, people (consumers too) should really not use passwords as their last line of defense.
xyst 5y ago
O&G was temporarily brought to its knees by a single password compromise. This is sad. All of their money and they can’t even implement basic edge security protocols.
gerdesj 5y ago
If your budget is tight you might consider the following for MF for your VPNs: pfSense VM at work with a single interface and an OpenVPN server listening with a server cert minted from your AD CA - this is your VPN concentrator. You get your OpenVPN server to hand off auth to RADIUS which is a DC with the NPS service installed. Mint SSL certs for all your domain joined users and computers via your AD CA. Investigate this parameter: cryptoapicert "SUBJ:" on your OpenVPN clients. All of this stuff is documented in various places and I've dropped most of the clues! You can actually fold in more factors with this. So the PC/user must have a SSL cert minted by the AD CA and a password, and a OTP via something like Google authenticator. I'd start with short lived (say one month) AD minted SSL certs and then decide whether you need more factors. Remember that a domain joined PC doesn't work very well at password change time if encryption and VPN connections rely on AD passwords. Now, once you allow your folks on site - what do you allow them to access? Get your firewall rules sorted.
- nszceta 5y ago
  This is hilariously complicated compared to deploying Wireguard VPN
  - gerdesj 5y ago
    Wireguard, OpenVPN, IPSEC etc are just ways of shuffling packets of data with encryption. What I'm describing is actually a really easy or at least really cheap way for an organisation to grab an extra factor when authenticating their VPNs. There are loads of howtos (most are completely wank but there are some decent ones) on minting AD SSL certs for users and computers. That's something you "have". You could simply use SSL certs or you could also insist on a username and maybe a password ("something you know"). You use RADIUS because that is a long, long standard way of meditating auth/auth to facility.
  - jcrawfordor 5y ago
    Compared to Wireguard with MFA? as far as I can tell Wireguard doesn't support MFA at all. Pure key auth is not acceptable in these contexts because of stolen keys and devices.
ineedasername 5y ago
The account was no longer in use at the time of the attack but could still be used to access Colonial’s network, he said. Very important to de-provision accounts when an employee leaves or some other factor renders it unnecessary.
gz5 5y ago
Password is the cause like a gun is the cause of a shooting. Networks are targets - both because their assets make them attractive to attack, and because a network is inherently vulnerable (since it is a set of connected nodes). So the cause is the network and the solution is to eliminate the network. Securely identified, properly authenticated, least privileged access, app-specific, ephemeral connections. Can such a connection still be breached? Of course. But it will be more difficult and it will be isolated, micro-segmented by definition, and unable to be leveraged to attack laterally. That said, networks makes things simple. And complexity is insecure. As soon as we can make the paradigm listed above as simple as networking, we will see a massive shift. We need it.
- walrus01 5y ago
  Password, bad. Password, allowing access to something with a very large 'blast radius', horrible. Systematic failure of compartmentalization and roles/access control within the organization's network. I do not agree that eliminating the network is the solution. Properly securing the network is. I could go into a 30 page essay on what might be considered within the category of 'proper', but suffice to say that organizations of this importance should have a serious infosec/netsec department.
  - anonymouse008 5y ago
    > I could go into a 30 page essay on what might be considered within the category of 'proper' I’d read that essay!
fidesomnes 5y ago
Imagine being the guy behind the screen when that shell popped. The heart racing paycheck bonus is palatable from here.
ClumsyPilot 5y ago
So we are in the territory where a script kiddy could download year old data leak from some random torrent tracker and use it to log in and shut down vital national infrastructure without even completing a Hacking course on Udemi.
- arthur_sav 5y ago
  The larger the attack surface, the easier it is. Large corps with thousands of employees and legacy infrastructure is usually not that hard to breach. While i haven't kept up with cyber security in a while i think the problem is largely the same. Security is not baked in, it's an afterthought at best and because of that there's shortage of talent.
- ed25519FUUU 5y ago
  Don’t worry Colonial is asking the government to go after these teenagers and that will finally fix the root of the issue.
  - gscott 5y ago
    Can't be too hard just look for the only 15 year old with a new G5 jet.
    - mcintyre1994 5y ago
      They’ll just tell you they sold their dogecoin though :)
HerbsMan 5y ago
Using passwords instead of certs in VPN that provides access to internal network is just to say it clearly stupid. Apparently morons working there, so, they deserved for it.
loonster 5y ago
Not everything needs to be connected to the internet. They own right of ways for a pipeline. Go bury some fiber along the pipe. All critical company communications can now be handled internally.
- ed25519FUUU 5y ago
  This is a good idea. Is it possible to require all critical infrastructure to be disconnected from the WAN?
  - bobthepanda 5y ago
    It isn’t if you want remote monitoring, probably. So doing that could raise labor costs.
- rurban 5y ago
  But than you have the famous office manager, who decides to shutdown all ops because his windows PC is virus infected. All critical infrastructure depends on one overly stupid manager.
olivermarks 5y ago
'In the wake of the attack on his company, Blount said he would like the U.S. government to go after hackers who have found safe haven in Russia. “Ultimately the government needs to focus on the actors themselves. As a private company, we don’t have a political capability of shutting down the host countries that have these bad actors in them.”' says it all, regardless of nation state
- vijayr02 5y ago
  Reagan gave green light to sabotage of Soviet oil pipeline (2004) https://news.ycombinator.com/item?id=27402062 https://news.ycombinator.com/item?id=27402062 To paraphrase Clausewitz, economic war is war by other means. > "In order to disrupt the Soviet gas supply, its hard currency earnings from the West, and the internal Russian economy, the pipeline software that was to run the pumps, turbines and valves was programmed to go haywire, after a decent interval, to reset pump speeds and valve settings to produce pressures far beyond those acceptable to pipeline joints and welds," Reed writes. "The result was the most monumental non-nuclear explosion and fire ever seen from space," he recalls, adding that US satellites picked up the explosion. Reed said that the blast occurred in the summer of 1982. "While there were no physical casualties from the pipeline explosion, there was significant damage to the Soviet economy," he writes. "Its ultimate bankruptcy, not a bloody battle or nuclear exchange, is what brought the Cold War to an end. In time the Soviets came to understand that they had been stealing bogus technology, but now what were they to do? By implication, every cell of the Soviet leviathan might be infected. They had no way of knowing which equipment was sound, which was bogus. All was suspect, which was the intended endgame for the entire operation." Seems familiar to our predicament today, no?
  - jcranmer 5y ago
    I feel like I should point out that this claim comes from an autobiographical book that has never been corroborated by any other source. When the claim involves "the most monumental non-nuclear explosion and fire ever seen from space," the lack of other sources is especially suspicious. I personally find the claim that the Soviets were using SCADA control systems in 1982 somewhat suspicious as well.
deleted 5y ago
[deleted]
blondie9x 5y ago
This should not be this easy. Really?
mac-chaffee 5y ago
> Soon after the attack, Colonial embarked on an exhaustive examination of the pipeline, tracking 29,000 miles on the ground and through the air to look for visible damage. Is that an extra zero? That's more than the Earth's circumference...
mellosouls 5y ago
The hack that took down the largest fuel pipeline in the U.S. and led to shortages across the East Coast was the result of a single compromised password, according to a cybersecurity consultant who responded to the attack. This is kind of like blaming unrecoverable data on the employee that accidentally deleted it - the problems are much bigger than a single compromised password (with the implied single-employee-blaming) and the credibility of reporters or security consultants proposing such causes risk being undermined with these kinds of simplifications.
- goodoldneon 5y ago
  Depends how you read it. When I see “was the result of a compromised password” I think “they should’ve used something mitigates the risk of compromised passwords”. I don’t think it’s blaming the employee but rather blaming the company’s fragile security model. Everyone should use MFA on as many things as possible
rurban 5y ago
The headline is only to drive home this bullshit narrative: > It was the first time Colonial had shut down the entirety of its gasoline pipeline system in its 57-year history, Blount said. “We had no choice at that point,” he said. “It was absolutely the right thing to do. At that time, we had no idea who was attacking us or what their motives were.” So they saw a windows ransomware screen on their windows office computers. Even a known ransomware, with a known decrypter they could not find with a simple search on their android/IPhone. So they decided it's a high probability that the ransomware will jump from the office PCs to the non-Windows internal pipeline control system. Foreign terrorists. Bullshit. As we later found out, they decided to shutdown the east coast infrastructure only because they could not calculate the exact gallons delivered to each customer via their office PCs. Rounding would have lead to real money loss apparently. FYI In the real world everybody else does rounding and estimation. The sensor data is for verification and control.
- spuz 5y ago
  > As we later found out, they decided to shutdown the east coast infrastructure only because they could not calculate the exact gallons delivered to each customer via their office PCs. What is the source for this? As far as I understood, they shutdown all their computer systems to prevent the malware from spreading to them which in most cases is the correct response.

240 points by 1cvmask 5y ago

https://archive.is/p2XY1 https://archive.is/p2XY1

> The hack that took down the largest fuel pipeline in the U.S. This needs to be remembered: the pipeline was okay all the time. It is accounting software that was taken down. A conspiracy theorist in me thinks that most of the PR activity around the case was (and is) intending to paint this fact over.

upofadown 5y ago

The big issue here is that a single access could so easily be spun up into complete access. In a large company you have to assume that even the highest wall will eventually be breached. A current employee will eventually provide access to someone up to no good. Or that current employee will themselves get up to no good. Corporate messaging is a good example of this sort of thing. Completely unencrypted inside the wall. Tons of login information just left laying around in old plaintext messages. Your security should not be all or nothing...

vlan0 5y ago

When will people realize we need to stop implementing security in this manner? The use of passwords and perimeter security should be DEAD. https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-171r2.pdf https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.S... https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-207.pdf https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.S...

lbriner 5y ago

As others have said, MFA would be a decent tool but not all attacks are against VPN so it wouldn't always be so useful. As everyone always teaches, security has to be in-depth and the one largely simple control that is very often missing is segmentation of the system. Being able to attack a single department is annoying but unlikely to be considered fatal, however, I think that even savvy IT ops people do not really understand vlans, using switches and air-gaps to keep things apart and really concentrating on any single-points of ingress like Sharepoint sites or whatever although most problems seem to spread across the network layer rather than bridging machines over HTTP. Also, even some simple controls like basic local backup, even to a crappy SAN box in your server room would at least enable you to stick your fingers up at ransomware.

lumost 5y ago

At this point I either use saved passwords or need to rotate the passwords on every request via password reset. Saved passwords are also an attack vector and as such my corporate laptop disables all mechanisms for saving passwords. I think it's time to make pin + 2 factor the standard over passwords.

rdiddly 5y ago

Compromised password of an inactive account. Do regular audits and deactivate unused accounts, people! At the very least that takes care of the "disgruntled former employee" scenario. There's a lot of virtual ink in this article devoted to the "how were the credentials compromised" question. I understand the curiosity, but it's mostly unknowable and therefore not too interesting. It's like asking "Why did it rain on the day I didn't bring my umbrella?" Is that the question? "Why did it rain?" The interesting part is the point at which Colonial still had control and gave it up, which was when that employee left or whatever happened, and they didn't revoke those credentials. I keep having a thought about how this account belonged to a fired employee who was not only bad at passwords, but bad at this job too. I have no idea if that's the case, as that's generally confidential info. Kinda funny to think of though.

nabla9 5y ago

In this and age, if companies are not giving employees FIDO2 security keys it's just incompetence.

arkadiyt 5y ago
The account didn't have any kind of 2fa - even SMS would have stopped this breach. Obviously hardware keys are the gold standard but I wouldn't trivialize the amount of work it takes to roll that out to an organization.
- Godel_unicode 5y ago
  I would, it's pretty trivial (I have done it for multiple organizations of different sizes/technical competencies). Especially for something like a VPN; at this point if you use a VPN solution which doesn't support MFA it almost certainly has other security issues as well.
- ineedasername 5y ago
  I have a family member who had a very junior role at a big finance company 15 years ago and they used hardware keys. Seems like national infrastructure should require similar levels of security.
- qbasic_forever 5y ago
  For any sysadmins or future sysadmins reading this though, do NOT put your service's 2FA on SMS. Do not use SMS for anything other than sending your grandparents a happy birthday message. It is trivially broken with a sim card hijack and we have seen many high profile incidents of people being hacked precisely through this vector.
  - rossdavidh 5y ago
    ...and yet, it would have been better than this setup.
- kevin_thibedeau 5y ago

1cvmask 5y ago

They failed to implement multi-factor authentication on their VPNs. This is one of the most common attack vectors. VPNs by themselves are not enough. Over two thirds of corporate attacks with VPNs have been through compromised static credentials. There are numerous ways of integrating VPNs with MFA ranging from a RADIUS integration to a SAML one. At Saas Pass the admin can choose and implement up to 15 different methods of MFA to VPNs ranging from FIDO keys to push approval to scanning encrypted barcodes to SMS (not recommended) etc... Disclaimer: Worked on the design of some of the mfa interfaces for the SAML based VPNs and the access control policies.

jl2718 5y ago
Personally I think RADIUS is a terrible idea. If that thing is compromised, or even the network or name service to connect to it, the attacker gets everything. Cert-based auth gives every client the ability to independently authorize every login without any ability to learn or compromise credentials. And there are lots of cert-based MFAs that require nothing more than a few lines in an Apache config, or even handled through the operating system to set up a vpn handshake. One of my biggest contributions at one place I worked was to dismantle their radius auth and replace it with token cert auth. It’s hard to write about that on a resume, because people tend to be more impressed with complexity than simplicity.
- er4hn 5y ago
  "Upgraded legacy, obsolete, AAA service with modern and standards compliant solution" And if it is ever asked about, boy do you have some great stories about upgrading from a system whose protocol name referred to dial in modems and used MD5 to obfuscate passwords.
  - qazxcvbnmlp 5y ago
    Thank you. Not the op but I have trouble coming up with this stuff for my own resume.
g_p 5y ago
One issue in the critical infrastructure space is that many of these MFA solutions are less robust and resilient than no MFA. Anything based on calling out to an external service (Duo, for example) is likely to be less resilient [1] than the static, no-external-dependencies system they use for plain password Auth. Clearly moving towards complex systems like RADIUS and SAML introduce even more moving parts that sit with interconnects between trusted and untrusted, internal and external components. That becomes a point of failure in the eyes of infrastructure companies. The challenge here seems to be getting the IT world to understand the focus on availability in critical infrastructure - historically, the focus has been on availability first, integrity second, confidentiality third. You could argue this shouldn't apply to VPN interfaces in to manage systems, but given the external dependencies of these systems, they feel far less likely to remain available than the kind of solutions the sector expects. On a separate note though, critical infrastructure need to re-think their budgets to pay for their increased availability needs, given most of the commercial solutions are third party hosted "as a service" offerings. [1] https://status.duo.com/history?page=2 https://status.duo.com/history?page=2

3pt14159 5y ago

Yes, MFA would have helped here, but I think it's besides the point. I've worked on a lot of projects, including one for a major stock exchange, and I've never seen perfect security. It's too slow. Even if you had MFA for the VPN, it doesn't stop local malware from being able to use the foothold to gain access to the network once the MFA is cleared. I'm not saying that MFA is useless, just that these types of things are very hard to get right and we only hear about the times the attackers got through. "Oh they should have done X" where X is any one of an increasingly long list of possible things they should have done ranging from security headers on their website, to MFA, to auditing their third party libraries, to locking down their development machines and networks (during Covid, ha!) it's never ending. The very best at it get by without total calamity, but the long tail of organizations just don't have the right staff, focus, or resources to do it. Edit: I'm not saying it's right! I don't think it is right, and I've worked with governments to bring in regulations in certain industries, but to characterize this all on one failure kinda misses the point. There is a broader, harder problem that I'm not sure is even solvable.

genmud 5y ago
Going to have to hard disagree there. MFA and addressing password reuse is table stakes in 2021. Like this is literally the first item I would do as a CISO in any company. MFA is among the core security items like running endpoint protection, updating your software or filtering spam. Bucketing it with things like CORS or auditing third party libraries is either disingenuous or extremely naive.
- 3pt14159 5y ago
  Look, MFA for access to code repositories or a cloud is one thing, but if you think that MFA is "table stakes" for a VPN I don't know what to tell you other than "then very few orgs are doing table stakes with their VPNs." These types of arguments are so hard to have over HN, because people with actual experience across all sorts of organizations can't explicitly say what they have seen and where because of NDAs and we all have our own viewpoints and individual experience. Bucketing MFA into one broad concept is as disingenuous as calling out security headers as being as benign as CORS. Does your org have MFA on SSH? On email? On the code repo? On the Slack login? On the VPN(s)? Etc. Etc. I have yet to see a single org that had MFA on everything and blaming a lack of MFA on the VPN misses the point. If they'd had it there, the attackers would have hit elsewhere. Most developers install random software and browser toolbars that they'll never have time to check and connect directly to their companies servers, repos, and communication tools. Pretending like MFA on a VPN would have magically solved everything is a pipedream. There are too many ways in, and every time someone gets in the response from the software community is always "oh why didn't they do X" while ignoring the larger truth: There are too many threat vectors and getting them all right, at all times, is nigh on impossible.

dmitrygr 5y ago

I wonder if a useful counter-measure would be slower internet connectivity it such places. It would be very hard to exfiltrate 100GB at 8kbit speeds, while 8kbit is plenty for simple few-byte commands one would need to manage a pipeline.

goodpoint 5y ago
It is, and data diodes are even better.
scoopertrooper 5y ago
From what I understand, the hackers never got anywhere near the control systems. The systems they compromised were related to billing and probably the company's sharepoint and emails (hard to think what else the could make up the balance of the 100GB outside of the monitoring of the control systems). You could still slow down attacks like this by limiting bandwidth (though 8kbits might be painful for document sharing). However, if you limited users a _slightly_ more reasonable 1 megabit per second, they'd be able to complete their exfiltration in within a little over 9 days. Keep in mind, Colonial only learned of the attack after the ransom demand was made!
- rapjr9 5y ago
  The article says the ransomware message appeared on a computer in the control room. That seems strange, do the pipeline operators adjust fuel flow based on how much cash is coming in? Is this a common practice? Or maybe it was not the billing system that was compromised? I guess it is possible billing would be used to predict required flow rate in some way, but that means any hack of the internet connected billing system has a direct route to controlling fuel flow in the pipeline, which seems rather dangerous. Post some fake orders and suddenly there are millions of gallons of fuel wherever you say they should be. Does the pipeline also run in reverse? If not it then that fuel could be stranded.
  - cesarb 5y ago
    > The article says the ransomware message appeared on a computer in the control room. That seems strange That computer was probably their "office" computer, which they would use for things like email. From what I understood from the article ("[...] an employee in Colonial’s control room saw a ransom note demanding cryptocurrency appear on a computer just before 5 a.m. The employee notified an operations supervisor who immediately began to start the process of shutting down the pipeline, [...]. By 6:10 a.m., the entire pipeline had been shut down"), the supervisor made the sensible decision of immediately starting an orderly shutdown of the pipeline, before the thing propagated to their control computers; as can be seen by the timing, doing an orderly shutdown takes a while.

vmception 5y ago

Password found in other leaks No multi factor authentication Wants government to go after people Government should go after Colonial

ineedasername 5y ago
Sure, penalties for negligent companies might help, but if someone leaves the front door open and gets robbed, you still go after the thieves too. Although the idea that the east coast pipeline infrastructure could be taken down by the rough equivalent of keeping the door open it pretty terrifying in its own right.
- alexanderdmitri 5y ago
  The door was closed and locked, but they a) didn't revoke an old key that got tied to it and b) didn't attempt to verify it was expected person using it when it was tried on the door. Making the distinction because I don't believe they were explicitly inviting people to rob them as much as they didn't take sufficient precautions in a bad neighborhood.
  - gwbas1c 5y ago
    Honestly, we can't let basic security turn into an arms race. It's getting harder and harder to maintain good password practice. Part of good security also needs to involve proactively going after the criminals. Anyone willing to break a window can break into my home. Internet security needs to be the same way: Secure enough to keep the honest people honest, and government agencies need to track down the hackers and imprison them. We can't expect normal people to perfectly implement complicated security procedures all the time, nor can we have pervasive technology and only run it with the world's smartest and most perfect people.
    - ridaj 5y ago
      Of course there needs to be consequences for thieves but that doesn't absolve people (particularly in critical infrastructure) from the responsibility to implement reasonable protections. If everyone keeps their doors unlocked in town and hundreds of burglaries occur as a result, the police should do something about it, but they may not be staffed to go after everyone. At some point they're going to be telling people to lock their houses please. "Password practice" is thankfully becoming less and less relevant as hardware security tokens become more widespread, affordable and reliable. IMO there's zero excuse for a critical infrastructure operator not to use 2FA at this point. Whatever the password policy is, people (consumers too) should really not use passwords as their last line of defense.

xyst 5y ago

O&G was temporarily brought to its knees by a single password compromise. This is sad. All of their money and they can’t even implement basic edge security protocols.

gerdesj 5y ago

If your budget is tight you might consider the following for MF for your VPNs: pfSense VM at work with a single interface and an OpenVPN server listening with a server cert minted from your AD CA - this is your VPN concentrator. You get your OpenVPN server to hand off auth to RADIUS which is a DC with the NPS service installed. Mint SSL certs for all your domain joined users and computers via your AD CA. Investigate this parameter: cryptoapicert "SUBJ:" on your OpenVPN clients. All of this stuff is documented in various places and I've dropped most of the clues! You can actually fold in more factors with this. So the PC/user must have a SSL cert minted by the AD CA and a password, and a OTP via something like Google authenticator. I'd start with short lived (say one month) AD minted SSL certs and then decide whether you need more factors. Remember that a domain joined PC doesn't work very well at password change time if encryption and VPN connections rely on AD passwords. Now, once you allow your folks on site - what do you allow them to access? Get your firewall rules sorted.

nszceta 5y ago
This is hilariously complicated compared to deploying Wireguard VPN
- gerdesj 5y ago
  Wireguard, OpenVPN, IPSEC etc are just ways of shuffling packets of data with encryption. What I'm describing is actually a really easy or at least really cheap way for an organisation to grab an extra factor when authenticating their VPNs. There are loads of howtos (most are completely wank but there are some decent ones) on minting AD SSL certs for users and computers. That's something you "have". You could simply use SSL certs or you could also insist on a username and maybe a password ("something you know"). You use RADIUS because that is a long, long standard way of meditating auth/auth to facility.
- jcrawfordor 5y ago
  Compared to Wireguard with MFA? as far as I can tell Wireguard doesn't support MFA at all. Pure key auth is not acceptable in these contexts because of stolen keys and devices.

ineedasername 5y ago

The account was no longer in use at the time of the attack but could still be used to access Colonial’s network, he said. Very important to de-provision accounts when an employee leaves or some other factor renders it unnecessary.

gz5 5y ago

Password is the cause like a gun is the cause of a shooting. Networks are targets - both because their assets make them attractive to attack, and because a network is inherently vulnerable (since it is a set of connected nodes). So the cause is the network and the solution is to eliminate the network. Securely identified, properly authenticated, least privileged access, app-specific, ephemeral connections. Can such a connection still be breached? Of course. But it will be more difficult and it will be isolated, micro-segmented by definition, and unable to be leveraged to attack laterally. That said, networks makes things simple. And complexity is insecure. As soon as we can make the paradigm listed above as simple as networking, we will see a massive shift. We need it.

walrus01 5y ago
Password, bad. Password, allowing access to something with a very large 'blast radius', horrible. Systematic failure of compartmentalization and roles/access control within the organization's network. I do not agree that eliminating the network is the solution. Properly securing the network is. I could go into a 30 page essay on what might be considered within the category of 'proper', but suffice to say that organizations of this importance should have a serious infosec/netsec department.
- anonymouse008 5y ago
  > I could go into a 30 page essay on what might be considered within the category of 'proper' I’d read that essay!

fidesomnes 5y ago

Imagine being the guy behind the screen when that shell popped. The heart racing paycheck bonus is palatable from here.

ClumsyPilot 5y ago

So we are in the territory where a script kiddy could download year old data leak from some random torrent tracker and use it to log in and shut down vital national infrastructure without even completing a Hacking course on Udemi.

arthur_sav 5y ago
The larger the attack surface, the easier it is. Large corps with thousands of employees and legacy infrastructure is usually not that hard to breach. While i haven't kept up with cyber security in a while i think the problem is largely the same. Security is not baked in, it's an afterthought at best and because of that there's shortage of talent.
ed25519FUUU 5y ago
Don’t worry Colonial is asking the government to go after these teenagers and that will finally fix the root of the issue.
- gscott 5y ago
  Can't be too hard just look for the only 15 year old with a new G5 jet.
  - mcintyre1994 5y ago
    They’ll just tell you they sold their dogecoin though :)

HerbsMan 5y ago

Using passwords instead of certs in VPN that provides access to internal network is just to say it clearly stupid. Apparently morons working there, so, they deserved for it.

loonster 5y ago

Not everything needs to be connected to the internet. They own right of ways for a pipeline. Go bury some fiber along the pipe. All critical company communications can now be handled internally.

ed25519FUUU 5y ago
This is a good idea. Is it possible to require all critical infrastructure to be disconnected from the WAN?
- bobthepanda 5y ago
  It isn’t if you want remote monitoring, probably. So doing that could raise labor costs.
rurban 5y ago
But than you have the famous office manager, who decides to shutdown all ops because his windows PC is virus infected. All critical infrastructure depends on one overly stupid manager.

olivermarks 5y ago

'In the wake of the attack on his company, Blount said he would like the U.S. government to go after hackers who have found safe haven in Russia. “Ultimately the government needs to focus on the actors themselves. As a private company, we don’t have a political capability of shutting down the host countries that have these bad actors in them.”' says it all, regardless of nation state

vijayr02 5y ago
Reagan gave green light to sabotage of Soviet oil pipeline (2004) https://news.ycombinator.com/item?id=27402062 https://news.ycombinator.com/item?id=27402062 To paraphrase Clausewitz, economic war is war by other means. > "In order to disrupt the Soviet gas supply, its hard currency earnings from the West, and the internal Russian economy, the pipeline software that was to run the pumps, turbines and valves was programmed to go haywire, after a decent interval, to reset pump speeds and valve settings to produce pressures far beyond those acceptable to pipeline joints and welds," Reed writes. "The result was the most monumental non-nuclear explosion and fire ever seen from space," he recalls, adding that US satellites picked up the explosion. Reed said that the blast occurred in the summer of 1982. "While there were no physical casualties from the pipeline explosion, there was significant damage to the Soviet economy," he writes. "Its ultimate bankruptcy, not a bloody battle or nuclear exchange, is what brought the Cold War to an end. In time the Soviets came to understand that they had been stealing bogus technology, but now what were they to do? By implication, every cell of the Soviet leviathan might be infected. They had no way of knowing which equipment was sound, which was bogus. All was suspect, which was the intended endgame for the entire operation." Seems familiar to our predicament today, no?
- jcranmer 5y ago
  I feel like I should point out that this claim comes from an autobiographical book that has never been corroborated by any other source. When the claim involves "the most monumental non-nuclear explosion and fire ever seen from space," the lack of other sources is especially suspicious. I personally find the claim that the Soviets were using SCADA control systems in 1982 somewhat suspicious as well.

deleted 5y ago

[deleted]

blondie9x 5y ago

This should not be this easy. Really?

mac-chaffee 5y ago

> Soon after the attack, Colonial embarked on an exhaustive examination of the pipeline, tracking 29,000 miles on the ground and through the air to look for visible damage. Is that an extra zero? That's more than the Earth's circumference...

mellosouls 5y ago

The hack that took down the largest fuel pipeline in the U.S. and led to shortages across the East Coast was the result of a single compromised password, according to a cybersecurity consultant who responded to the attack. This is kind of like blaming unrecoverable data on the employee that accidentally deleted it - the problems are much bigger than a single compromised password (with the implied single-employee-blaming) and the credibility of reporters or security consultants proposing such causes risk being undermined with these kinds of simplifications.

goodoldneon 5y ago
Depends how you read it. When I see “was the result of a compromised password” I think “they should’ve used something mitigates the risk of compromised passwords”. I don’t think it’s blaming the employee but rather blaming the company’s fragile security model. Everyone should use MFA on as many things as possible

rurban 5y ago

The headline is only to drive home this bullshit narrative: > It was the first time Colonial had shut down the entirety of its gasoline pipeline system in its 57-year history, Blount said. “We had no choice at that point,” he said. “It was absolutely the right thing to do. At that time, we had no idea who was attacking us or what their motives were.” So they saw a windows ransomware screen on their windows office computers. Even a known ransomware, with a known decrypter they could not find with a simple search on their android/IPhone. So they decided it's a high probability that the ransomware will jump from the office PCs to the non-Windows internal pipeline control system. Foreign terrorists. Bullshit. As we later found out, they decided to shutdown the east coast infrastructure only because they could not calculate the exact gallons delivered to each customer via their office PCs. Rounding would have lead to real money loss apparently. FYI In the real world everybody else does rounding and estimation. The sensor data is for verification and control.

spuz 5y ago
> As we later found out, they decided to shutdown the east coast infrastructure only because they could not calculate the exact gallons delivered to each customer via their office PCs. What is the source for this? As far as I understood, they shutdown all their computer systems to prevent the malware from spreading to them which in most cases is the correct response.