5 ms·

How will this even begin to be remediated (the broader hack that is coming to light right now)? It seems like malicious actors had unrestricted access to almos

by blhack 6y ago

How will this even begin to be remediated (the broader hack that is coming to light right now)? It seems like malicious actors had unrestricted access to almost every major computer system in the US Government, and now possibly microsoft itself as well? How are these people ever going to be able to trust any of this equipment ever again? This just seems unbelievably catastrophic.

mr_overalls 6y ago
It could be incredibly expensive to clean up. Remediation and recovery for most threats involves OS/app reinstallation, perhaps restoring from backups and images. However, if your threat is a sophisticated state actor based out of Russia, it's hard to rule out that they're got hooks in your server's firmware, that they've corrupted your backups as well, etc, etc. One wonders how Russia could exploit the systems they've penetrated. Brick every gov't system on Jan 20th? Shut down SCADA systems? It's a cybersecurity nightmare.
- panarky 6y ago
  They've been inside since March, so any pre-breach backups and images are probably too old to restore.
  - mrtnmcc 6y ago
    Last stable release was r2016.
    - x86_64Ubuntu 6y ago
      I think he's talking about the hackers having been inside the victimized systems.
- unclekev 6y ago
  > Remediation and recovery for most threats involves OS/app reinstallation Except for all those SolarWinds admins arguing that doing a simple scan and infected binary removal is enough and then moving on and anything more is "overreacting" I feel sorry for all these people who are stuck working with such inflexible risk assessment/ITIL processes who are now trying to justify not taking any action because "SolarWinds said everything is ok"
  - ownagefool 6y ago
    It's quite a mess indeed. There's obviously a contemporary movement that all your systems should be rebuildable by code, which would make getting the systems back into a trusted state (assuming you trust other layers / your code) a lot easier. Obviously this doesn't help if your data is already messed up, if firmwares are hacked, and if your code itself hasn't had te rigour to be trusted, but it's a hell of a lot better position than "scan, remove, forget".
_trampeltier 6y ago
https://webcache.googleusercontent.com/search?q=cache:2CZ1Lu0Qd00J:https://www.solarwinds.com/es/company/customers+&cd=1&hl=en&ct=clnk&gl=us https://webcache.googleusercontent.com/search?q=cache:2CZ1Lu... Microsoft won't be the last company ..
- blhack 6y ago
  Am I confused about something? How is everybody not absolutely running around frantically ripping ethernet cables out of patch panels right now in every single one of these companies? I mean...the smart controllers on the HVAC systems in these companies have to be replaced don't they? The smart locks, everything IoT, everything with a network interface in it at this point has to be assumed compromised. This seems like by far the worst cyber security incident of all time.
  - webmaven 6y ago
    > I mean...the smart controllers on the HVAC systems in these companies have to be replaced don't they? Did you recently binge-watch Mr. Robot?
    - blhack 6y ago
      That was a nod to an actual hack that happened back in 2014: https://krebsonsecurity.com/2014/02/target-hackers-broke-in-via-hvac-company/ https://krebsonsecurity.com/2014/02/target-hackers-broke-in-... And my comment was also a nod to that prominent hack. The discussion back then revolved a lot around the idea that Target had done a really good job of hardening most of their network, but then allowed a smart HVAC controller onto it. What seemed at the time like something minor (I believe it was a remote diagnostic device or something like that) is what the intruders used to gain access.
      - XMPPwocky 6y ago
        Your article seems to directly contradict the idea that any sort of IoT / "smart" device was involved: > Fazio Mechanical Services just issued an official statement through a PR company, stating that its “data connection with Target was exclusively for electronic billing, contract submission and project management.”
swayvil 6y ago
Open the code to public auditing. It's the only way. A million eyes will make short work of the cleanup.
- acdha 6y ago
  If you think that’s sufficient, you have some reading to do. Start with Ken Thompson: http://users.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf http://users.ece.cmu.edu/~ganger/712.fall02/papers/p761-thom... Now ask about all of the things opening source wouldn’t affect: beyond compilers, modern devices have a lot of software running in firmware which can alter data. Proving that every component involved in the process hasn’t been subverted is a massive undertaking.
- rapsey 6y ago
  This is a dumb and harmful myth. Very, very few bother reading the source.
  - tovej 6y ago
    This is absolutely not a dumb myth. It's obviously true that open source is not sufficient to good software quality, or even necessary, but it does correlate, especially for open source projects with many users. I often read through the libraries I use in projects (if their source is available). And if I find errors or shortcomings I will write an issue about it.
gbrown 6y ago
Also, the rest of us use products they’ve designed to be basically unsecurable against them. The feudal model of security only works if the overlords are trustworthy and competent. We’ve known they aren’t trustworthy for a long time, but this shows the other side of the coin.
- joe_the_user 6y ago
  Plus the feudal model only works if there's a definite hierarchy. The "everyone gets to get in everyone else's business" model clearly is a disaster. This disaster.
  - deleted 6y ago
    [deleted]
- OliverGilan 6y ago
  I've never heard of the feudal model with security. Can you explain?
  - gbrown 6y ago
    The idea is that security is hard and expensive, so we serfs surrender ourselves to feudal overlords (Google, Microsoft, Facebook) in exchange for protection.
    - Red_Leaves_Flyy 6y ago
      This model has been bitterly fought since the beginning. Problem is the underdogs have never had the money, and governmental support to manufacture consent in the masses.
      - ASalazarMX 6y ago
        What are the alternatives? Surely we can't design our own systems from scratch and outdo FAANGs in terms of security, and any underdog trying to change the status quo will end becoming a feudal lord itself.
lawnchair_larry 6y ago
This is somewhat routine actually. Microsoft, and most other major tech companies, have been “hacked” many times. Note that being hacked isn’t a binary state. What matters is what they were able to obtain. It could range from full compromise of the C-suite and domain admin, to phishing some marketing employee with no access to anything interesting. If anything, you should be afraid of companies who haven’t been hacked. It most likely means they’re either irrelevant, or they have been hacked and don’t know it yet. This isn’t even the first time they’ve been hacked by Russians. It’s honestly not a big deal.
- partingshots 6y ago
  Downplaying the problem doesn’t magically make it go away you know.
  - lawnchair_larry 6y ago
    I’m not sure what the purpose of your comment is. As someone who has been on the inside of these attacks, I’m just saying, what probably sounds earth shattering to most people is just a slightly more interesting Thursday for us. My expectations for security have been calibrated to be unfazed by yet another one. Honestly, it’s actually a little refreshing to see something slightly novel (although this isn’t actually that novel).
snarfy 6y ago
deleted
- wyldfire 6y ago
  This is pretty obnoxious and doesn't belong on HN. Millions of people live there. If we are somehow able to leave our humanity behind, the practical fact is that would lead to widespread annihilation.
  - snarfy 6y ago
    You're right and I removed my post.
badrabbit 6y ago
Incident response procedures exist to address this as does forensic analysis. But each org might fail at eradication (hardest phase of IR) and get reinfected. It is hard but doable imo
rk06 6y ago
As per the article, they used microsoft's cloud services i.e. azure for their attack, instead of breaking into microsoft's infra.
ackbar03 6y ago
I'm waiting for someone to say blockchain
- rbanffy 6y ago
  If all computers in the datacenter were mining bitcoin, the attacker wouldn't be able to use them for anything else.

by blhack 6y ago

mr_overalls 6y ago

It could be incredibly expensive to clean up. Remediation and recovery for most threats involves OS/app reinstallation, perhaps restoring from backups and images. However, if your threat is a sophisticated state actor based out of Russia, it's hard to rule out that they're got hooks in your server's firmware, that they've corrupted your backups as well, etc, etc. One wonders how Russia could exploit the systems they've penetrated. Brick every gov't system on Jan 20th? Shut down SCADA systems? It's a cybersecurity nightmare.

panarky 6y ago
They've been inside since March, so any pre-breach backups and images are probably too old to restore.
- mrtnmcc 6y ago
  Last stable release was r2016.
  - x86_64Ubuntu 6y ago
    I think he's talking about the hackers having been inside the victimized systems.
unclekev 6y ago
> Remediation and recovery for most threats involves OS/app reinstallation Except for all those SolarWinds admins arguing that doing a simple scan and infected binary removal is enough and then moving on and anything more is "overreacting" I feel sorry for all these people who are stuck working with such inflexible risk assessment/ITIL processes who are now trying to justify not taking any action because "SolarWinds said everything is ok"
- ownagefool 6y ago
  It's quite a mess indeed. There's obviously a contemporary movement that all your systems should be rebuildable by code, which would make getting the systems back into a trusted state (assuming you trust other layers / your code) a lot easier. Obviously this doesn't help if your data is already messed up, if firmwares are hacked, and if your code itself hasn't had te rigour to be trusted, but it's a hell of a lot better position than "scan, remove, forget".

_trampeltier 6y ago

https://webcache.googleusercontent.com/search?q=cache:2CZ1Lu0Qd00J:https://www.solarwinds.com/es/company/customers+&cd=1&hl=en&ct=clnk&gl=us https://webcache.googleusercontent.com/search?q=cache:2CZ1Lu... Microsoft won't be the last company ..

blhack 6y ago
Am I confused about something? How is everybody not absolutely running around frantically ripping ethernet cables out of patch panels right now in every single one of these companies? I mean...the smart controllers on the HVAC systems in these companies have to be replaced don't they? The smart locks, everything IoT, everything with a network interface in it at this point has to be assumed compromised. This seems like by far the worst cyber security incident of all time.
- webmaven 6y ago
  > I mean...the smart controllers on the HVAC systems in these companies have to be replaced don't they? Did you recently binge-watch Mr. Robot?
  - blhack 6y ago
    That was a nod to an actual hack that happened back in 2014: https://krebsonsecurity.com/2014/02/target-hackers-broke-in-via-hvac-company/ https://krebsonsecurity.com/2014/02/target-hackers-broke-in-... And my comment was also a nod to that prominent hack. The discussion back then revolved a lot around the idea that Target had done a really good job of hardening most of their network, but then allowed a smart HVAC controller onto it. What seemed at the time like something minor (I believe it was a remote diagnostic device or something like that) is what the intruders used to gain access.
    - XMPPwocky 6y ago
      Your article seems to directly contradict the idea that any sort of IoT / "smart" device was involved: > Fazio Mechanical Services just issued an official statement through a PR company, stating that its “data connection with Target was exclusively for electronic billing, contract submission and project management.”

swayvil 6y ago

Open the code to public auditing. It's the only way. A million eyes will make short work of the cleanup.

acdha 6y ago
If you think that’s sufficient, you have some reading to do. Start with Ken Thompson: http://users.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf http://users.ece.cmu.edu/~ganger/712.fall02/papers/p761-thom... Now ask about all of the things opening source wouldn’t affect: beyond compilers, modern devices have a lot of software running in firmware which can alter data. Proving that every component involved in the process hasn’t been subverted is a massive undertaking.
rapsey 6y ago
This is a dumb and harmful myth. Very, very few bother reading the source.
- tovej 6y ago
  This is absolutely not a dumb myth. It's obviously true that open source is not sufficient to good software quality, or even necessary, but it does correlate, especially for open source projects with many users. I often read through the libraries I use in projects (if their source is available). And if I find errors or shortcomings I will write an issue about it.

gbrown 6y ago

Also, the rest of us use products they’ve designed to be basically unsecurable against them. The feudal model of security only works if the overlords are trustworthy and competent. We’ve known they aren’t trustworthy for a long time, but this shows the other side of the coin.

joe_the_user 6y ago
Plus the feudal model only works if there's a definite hierarchy. The "everyone gets to get in everyone else's business" model clearly is a disaster. This disaster.
- deleted 6y ago
  [deleted]
OliverGilan 6y ago
I've never heard of the feudal model with security. Can you explain?
- gbrown 6y ago
  The idea is that security is hard and expensive, so we serfs surrender ourselves to feudal overlords (Google, Microsoft, Facebook) in exchange for protection.
  - Red_Leaves_Flyy 6y ago
    This model has been bitterly fought since the beginning. Problem is the underdogs have never had the money, and governmental support to manufacture consent in the masses.
    - ASalazarMX 6y ago
      What are the alternatives? Surely we can't design our own systems from scratch and outdo FAANGs in terms of security, and any underdog trying to change the status quo will end becoming a feudal lord itself.

lawnchair_larry 6y ago

This is somewhat routine actually. Microsoft, and most other major tech companies, have been “hacked” many times. Note that being hacked isn’t a binary state. What matters is what they were able to obtain. It could range from full compromise of the C-suite and domain admin, to phishing some marketing employee with no access to anything interesting. If anything, you should be afraid of companies who haven’t been hacked. It most likely means they’re either irrelevant, or they have been hacked and don’t know it yet. This isn’t even the first time they’ve been hacked by Russians. It’s honestly not a big deal.

partingshots 6y ago
Downplaying the problem doesn’t magically make it go away you know.
- lawnchair_larry 6y ago
  I’m not sure what the purpose of your comment is. As someone who has been on the inside of these attacks, I’m just saying, what probably sounds earth shattering to most people is just a slightly more interesting Thursday for us. My expectations for security have been calibrated to be unfazed by yet another one. Honestly, it’s actually a little refreshing to see something slightly novel (although this isn’t actually that novel).

snarfy 6y ago

deleted

wyldfire 6y ago
This is pretty obnoxious and doesn't belong on HN. Millions of people live there. If we are somehow able to leave our humanity behind, the practical fact is that would lead to widespread annihilation.
- snarfy 6y ago
  You're right and I removed my post.

badrabbit 6y ago

Incident response procedures exist to address this as does forensic analysis. But each org might fail at eradication (hardest phase of IR) and get reinfected. It is hard but doable imo

rk06 6y ago

As per the article, they used microsoft's cloud services i.e. azure for their attack, instead of breaking into microsoft's infra.

ackbar03 6y ago

I'm waiting for someone to say blockchain

rbanffy 6y ago
If all computers in the datacenter were mining bitcoin, the attacker wouldn't be able to use them for anything else.