10 ms·

So apparently Russian hackers were able to infiltrate the Office 365 accounts of multiple federal agencies. They were able to do to this by targeting one of th

So apparently Russian hackers were able to infiltrate the Office 365 accounts of multiple federal agencies. They were able to do to this by targeting one of the government's suppliers, a company called "SolarWinds" in Austin. The hackers were able to slip their software into a software update from SolarWinds over the summer. And get this: "SolarWinds says on its website that its customers include most of America’s Fortune 500 companies, all top ten U.S. telecommunications providers, all five branches of the U.S. military, the State Department, the National Security Agency, and the Office of President of the United States". Yikes.

mdgrech23 6y ago
curios as to how Russian hackers slipped their software into solar winds. sounds like a major breach.
- omarforgotpwd 6y ago
  If I had to guess, probably some kind of social engineering attack. Identify the supplier. Go on LinkedIn, look for employees of that supplier with a title that would imply sufficient privileges to enable the attack. Then get to know that person and target them personally.
  - o-__-o 6y ago
    Or become employed there and go for the long con. How many companies check in go vendor code or pip install from CI? Ripe targets for abuse
    - Teknoman117 6y ago
      Where I work we forbid this kind of behavior but you would not believe the kind of pushback we get for "making things too difficult".
  - finaliteration 6y ago
    I was the target of one of these during a recent internal pen test and got caught in it, despite being very technically savvy and aware of “normal” phishing techniques. The attack that was simulated in my case utilized convincing social engineering, spear phishing, domain spoofing, and malicious OAuth apps meant to look like an internal resource/service to gain access to sensitive material. It was very sophisticated and I’m glad I fell for it during a simulation rather than in a “real life” situation. It was a learning experience and a situation I’m way more paranoid about now. I could easily see admins and developers anywhere falling for it if they were specifically targeted.
    - threentaway 6y ago
      Organizations that are serious about security should not allow random OAuth apps. Both G Suite and O365 admins can restrict what OAuth apps are allowed.
      - finaliteration 6y ago
        Agreed. However some service providers (like GitHub) allow a user with admin access to approve any OAuth app the first time it’s “launched” if they are the ones initiating the request, rather than using a pre-defined allowlist approach which I believe is what Office 365 does. My wish would be for some sort of multi-person approval process rather than allowing anyone who is an admin to authorize an app. Even admins can be susceptible to a targeted and advanced attack. Also, many people (like myself before my own “failure”) simply aren’t aware of OAuth apps as a serious attack vector. Most remedial training around phishing campaigns covers things like fake login pages but not “An attacker has spoofed an internal domain and an OAuth app with your company’s name in it”.
      - marcosdumay 6y ago
        Domain spoofing takes care of any such restriction.
    - paranoidrobot 6y ago
      > The attack that was simulated in my case utilized convincing social engineering, spear phishing, domain spoofing, and malicious OAuth apps meant to look like an internal resource/service to gain access to sensitive material. This sounds like a post I saw on Reddit a few days ago. This person's IT organisation had been talking about migrating to Github Enterprise, they got an email saying that it had been rolled out from an internal IT mailbox to an OAuth application that had been pre-approved on their Github Organisation. For that particular scenario - if the org-admins have approved the OAuth application and are able to send mail from within the organisation - then it's probably game over anyway, since to approve the application they probably needed Admin rights anyway.
  - stainforth 6y ago
    Looks like LinkedIn is a big security concern, I won't miss it. Imagine if companies require that you're ~not~ on it.
    - croh 6y ago
      Not only linkedin but all job portals.
    - hn_throwaway_99 6y ago
      The security heads of the previous companies I worked at all obfuscated their LinkedIn profiles.
      - yalok 6y ago
        don’t post current work place until it’s over.
- netflixandkill 6y ago
  An insider got them access to a trusted signing certificate would be my first guess. PKI remains a pain in the ass and very few organizations do it well.
tomc1985 6y ago
Shit, SolarWinds is how MS does a lot of its electronic software distribution. I've received multiple things from MS SolarWinds download links.
sneak 6y ago
...and yet somehow people tell me I'm crazy when I demand that software not autoupdate without user intervention. Automatic updates are RCE vulnerabilities.
- drak0n1c 6y ago
  Election officials in two counties claimed that an automatic update to Dominion machines was to blame for their multi-hour outage during the election. Automatic updates are not only vulnerabilities but can undermine trust in elections. https://www.politico.com/news/2020/11/04/georgia-election-machine-glitch-434065 https://www.politico.com/news/2020/11/04/georgia-election-ma...
- HALtheWise 6y ago
  In this case, it doesn't appear to me that having the update be a manual process would have made it any easier to detect.
- arkadiyt 6y ago
  No one is arguing a compromise of the update server doesn't allow compromising all customers. They're arguing that the benefits out auto-updating software (especially consumer software) outweigh the risk of a compromised update server, which is generally true.
  - sneak 6y ago
    What about users who don't consent to that, who would prefer to manually update in a safe manner? Making the decision for a user is the issue. Consent to run software A on Monday is not consent to run unknown/unexamined software B on Tuesday. In the vast majority of cases, receiving userspace updates in a timely fashion is not worth this real-time remote access vulnerability.
- cookiemonster9 6y ago
  To be clear, given that one never knows if or when a provider has been compromised... is the plan to just not update? What if they were compromised before you initially obtained the software? There's not much that can stop attacks like this. Preventing lateral movement, escalation, exfiltration, detection, and remediation, among other things, would be the way to go.
rmrfstar 6y ago
Interesting that they're called SolarWinds. They make netflow analysis tools... kinda like Stellar Wind.
- VistaBrokeMyPC 6y ago
  I picked up on that too. I guess that means SolarWinds is a good target for nation-state level hacking
astura 6y ago
I believe it was an insider. I have personal experience delivering software/software updates to the USG. I'm actually baffled as to how something like this can happen without an insider. I've never had any slight sliver of concern over the security of our supply chains.
- Fordec 6y ago
  The said it involved a manual supply chain breach. Sounds like an insider with a USB drive.
- deathhand 6y ago
  I believe not everyone always checks the checksums. "Never ascribe to malice that which is adequately explained by incompetence"
  - DangitBobby 6y ago
    I'm sure malicious people are very grateful that this platitude is in circulation.
    - andrewflnr 6y ago
      Calling it a platitude is selling it short. Yes, it's a little strained in the context of people trying to deceive you, but the malicious are still vastly outnumbered by the incompetent.
      - DangitBobby 6y ago
        In general, I agree that most bad things happen by accident, but I think generally it's an unhelpful statement. You should be slightly more suspicious of someone claiming innocence when they clearly experience benefits from the consequences of their actions.
  - wyldfire 6y ago
    Windows checks the signature.
pimlottc 6y ago
This is obvious not good, but just to note, when they say "the Office of the President of the United States", they're most likely referring to the Executive Office of the President (EOP), which consists of a few thousand employee, most of whom are non-partisan career civil servants. It's a much broader group than what it sounds like. https://en.wikipedia.org/wiki/Executive_Office_of_the_President_of_the_United_States https://en.wikipedia.org/wiki/Executive_Office_of_the_Presid...
op03 6y ago
> Yikes Well thanks to working at a large org I can say the mountains of pointless conversations that anyone has to dig through to find anything useful is the biggest security feature. Not the security tech. I mean this is the govt we are talking about. Known for inefficiency and incompotence. The person or team going through that shitpile is probably at high risk of brain damage. Its easy to find something juicy happening in one dept, like looking down a random manhole in a giant sewage system. But to figure out the dynamics across the system is why govts end up employing tens of thousands of people. Which no nation state or group can match. If Singapore or Dubai or Switzerland leak thats a much simpler sewage system. But the US is operating at a totally different scale. Killing or manipulating a few ants wont change giant ant hill behavior other than in Star Wars movies. We are still in the stone age in how we deal with large data describing large systems. The list of large leaks and breaches will grow but look for/at impact, filtering out the noise from the sky is falling buffoon class.
- deleted 6y ago
  [deleted]
geoffpado 6y ago
SolarWinds also owns some SaaS companies that seem popular among small-to-midsized tech companies; Pingdom and Loggly are a few that I’ve seen at more than one job.
redis_mlc 6y ago
1) I was involved in improving the security scanning of SignalFX agents about a year ago (if there's a changelog, then you can figure out the month.) So SignalFX has at least basic processes in place now based, on results from our scanning. You will have to upgrade if you're using older packages and want the benefits of the new release processes. Note they were initially bewildered, until I pointed out that their entire client list could be pwned. Then they got on top of it. 2) Note that if you use multiple agents, it's possible to have up to 5 GB RAM used for agents. That means if you're using 8GB RAM containers/VMs in production to run java applications, they are starting with less than 2 GB RAM for application and heap. You might want to check on that. Upvote to thank me for the time invested in improving their agent packages.
teawrecks 6y ago
"I asked Putin if he hacked us and he said they didn't."
- dang 6y ago
  Could you please stop posting unsubstantive comments to Hacker News? You've been doing it a lot lately and we're trying for something different here. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

mdgrech23 6y ago

curios as to how Russian hackers slipped their software into solar winds. sounds like a major breach.

omarforgotpwd 6y ago
If I had to guess, probably some kind of social engineering attack. Identify the supplier. Go on LinkedIn, look for employees of that supplier with a title that would imply sufficient privileges to enable the attack. Then get to know that person and target them personally.
- o-__-o 6y ago
  Or become employed there and go for the long con. How many companies check in go vendor code or pip install from CI? Ripe targets for abuse
  - Teknoman117 6y ago
    Where I work we forbid this kind of behavior but you would not believe the kind of pushback we get for "making things too difficult".
- finaliteration 6y ago
  I was the target of one of these during a recent internal pen test and got caught in it, despite being very technically savvy and aware of “normal” phishing techniques. The attack that was simulated in my case utilized convincing social engineering, spear phishing, domain spoofing, and malicious OAuth apps meant to look like an internal resource/service to gain access to sensitive material. It was very sophisticated and I’m glad I fell for it during a simulation rather than in a “real life” situation. It was a learning experience and a situation I’m way more paranoid about now. I could easily see admins and developers anywhere falling for it if they were specifically targeted.
  - threentaway 6y ago
    Organizations that are serious about security should not allow random OAuth apps. Both G Suite and O365 admins can restrict what OAuth apps are allowed.
    - finaliteration 6y ago
      Agreed. However some service providers (like GitHub) allow a user with admin access to approve any OAuth app the first time it’s “launched” if they are the ones initiating the request, rather than using a pre-defined allowlist approach which I believe is what Office 365 does. My wish would be for some sort of multi-person approval process rather than allowing anyone who is an admin to authorize an app. Even admins can be susceptible to a targeted and advanced attack. Also, many people (like myself before my own “failure”) simply aren’t aware of OAuth apps as a serious attack vector. Most remedial training around phishing campaigns covers things like fake login pages but not “An attacker has spoofed an internal domain and an OAuth app with your company’s name in it”.
    - marcosdumay 6y ago
      Domain spoofing takes care of any such restriction.
  - paranoidrobot 6y ago
    > The attack that was simulated in my case utilized convincing social engineering, spear phishing, domain spoofing, and malicious OAuth apps meant to look like an internal resource/service to gain access to sensitive material. This sounds like a post I saw on Reddit a few days ago. This person's IT organisation had been talking about migrating to Github Enterprise, they got an email saying that it had been rolled out from an internal IT mailbox to an OAuth application that had been pre-approved on their Github Organisation. For that particular scenario - if the org-admins have approved the OAuth application and are able to send mail from within the organisation - then it's probably game over anyway, since to approve the application they probably needed Admin rights anyway.
- stainforth 6y ago
  Looks like LinkedIn is a big security concern, I won't miss it. Imagine if companies require that you're ~not~ on it.
  - croh 6y ago
    Not only linkedin but all job portals.
  - hn_throwaway_99 6y ago
    The security heads of the previous companies I worked at all obfuscated their LinkedIn profiles.
    - yalok 6y ago
      don’t post current work place until it’s over.
netflixandkill 6y ago
An insider got them access to a trusted signing certificate would be my first guess. PKI remains a pain in the ass and very few organizations do it well.

tomc1985 6y ago

Shit, SolarWinds is how MS does a lot of its electronic software distribution. I've received multiple things from MS SolarWinds download links.

sneak 6y ago

...and yet somehow people tell me I'm crazy when I demand that software not autoupdate without user intervention. Automatic updates are RCE vulnerabilities.

drak0n1c 6y ago
Election officials in two counties claimed that an automatic update to Dominion machines was to blame for their multi-hour outage during the election. Automatic updates are not only vulnerabilities but can undermine trust in elections. https://www.politico.com/news/2020/11/04/georgia-election-machine-glitch-434065 https://www.politico.com/news/2020/11/04/georgia-election-ma...
HALtheWise 6y ago
In this case, it doesn't appear to me that having the update be a manual process would have made it any easier to detect.
arkadiyt 6y ago
No one is arguing a compromise of the update server doesn't allow compromising all customers. They're arguing that the benefits out auto-updating software (especially consumer software) outweigh the risk of a compromised update server, which is generally true.
- sneak 6y ago
  What about users who don't consent to that, who would prefer to manually update in a safe manner? Making the decision for a user is the issue. Consent to run software A on Monday is not consent to run unknown/unexamined software B on Tuesday. In the vast majority of cases, receiving userspace updates in a timely fashion is not worth this real-time remote access vulnerability.
cookiemonster9 6y ago
To be clear, given that one never knows if or when a provider has been compromised... is the plan to just not update? What if they were compromised before you initially obtained the software? There's not much that can stop attacks like this. Preventing lateral movement, escalation, exfiltration, detection, and remediation, among other things, would be the way to go.

rmrfstar 6y ago

Interesting that they're called SolarWinds. They make netflow analysis tools... kinda like Stellar Wind.

VistaBrokeMyPC 6y ago
I picked up on that too. I guess that means SolarWinds is a good target for nation-state level hacking

astura 6y ago

I believe it was an insider. I have personal experience delivering software/software updates to the USG. I'm actually baffled as to how something like this can happen without an insider. I've never had any slight sliver of concern over the security of our supply chains.

Fordec 6y ago
The said it involved a manual supply chain breach. Sounds like an insider with a USB drive.
deathhand 6y ago
I believe not everyone always checks the checksums. "Never ascribe to malice that which is adequately explained by incompetence"
- DangitBobby 6y ago
  I'm sure malicious people are very grateful that this platitude is in circulation.
  - andrewflnr 6y ago
    Calling it a platitude is selling it short. Yes, it's a little strained in the context of people trying to deceive you, but the malicious are still vastly outnumbered by the incompetent.
    - DangitBobby 6y ago
      In general, I agree that most bad things happen by accident, but I think generally it's an unhelpful statement. You should be slightly more suspicious of someone claiming innocence when they clearly experience benefits from the consequences of their actions.
- wyldfire 6y ago
  Windows checks the signature.

pimlottc 6y ago

This is obvious not good, but just to note, when they say "the Office of the President of the United States", they're most likely referring to the Executive Office of the President (EOP), which consists of a few thousand employee, most of whom are non-partisan career civil servants. It's a much broader group than what it sounds like. https://en.wikipedia.org/wiki/Executive_Office_of_the_President_of_the_United_States https://en.wikipedia.org/wiki/Executive_Office_of_the_Presid...

op03 6y ago

> Yikes Well thanks to working at a large org I can say the mountains of pointless conversations that anyone has to dig through to find anything useful is the biggest security feature. Not the security tech. I mean this is the govt we are talking about. Known for inefficiency and incompotence. The person or team going through that shitpile is probably at high risk of brain damage. Its easy to find something juicy happening in one dept, like looking down a random manhole in a giant sewage system. But to figure out the dynamics across the system is why govts end up employing tens of thousands of people. Which no nation state or group can match. If Singapore or Dubai or Switzerland leak thats a much simpler sewage system. But the US is operating at a totally different scale. Killing or manipulating a few ants wont change giant ant hill behavior other than in Star Wars movies. We are still in the stone age in how we deal with large data describing large systems. The list of large leaks and breaches will grow but look for/at impact, filtering out the noise from the sky is falling buffoon class.

deleted 6y ago
[deleted]

geoffpado 6y ago

SolarWinds also owns some SaaS companies that seem popular among small-to-midsized tech companies; Pingdom and Loggly are a few that I’ve seen at more than one job.

redis_mlc 6y ago

1) I was involved in improving the security scanning of SignalFX agents about a year ago (if there's a changelog, then you can figure out the month.) So SignalFX has at least basic processes in place now based, on results from our scanning. You will have to upgrade if you're using older packages and want the benefits of the new release processes. Note they were initially bewildered, until I pointed out that their entire client list could be pwned. Then they got on top of it. 2) Note that if you use multiple agents, it's possible to have up to 5 GB RAM used for agents. That means if you're using 8GB RAM containers/VMs in production to run java applications, they are starting with less than 2 GB RAM for application and heap. You might want to check on that. Upvote to thank me for the time invested in improving their agent packages.

teawrecks 6y ago

"I asked Putin if he hacked us and he said they didn't."

dang 6y ago
Could you please stop posting unsubstantive comments to Hacker News? You've been doing it a lot lately and we're trying for something different here. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html