15 ms·

The Story of the SolarWinds Hack

305 points by bokchoi 5y ago

cyberlab 5y ago
> The routine update, it turns out, is no longer so routine Is there the rare case that we shouldn't update because the update could contain a malicious payload? If the update gets served over plaintext HTTP I would treat it as suspicious and may even block it from connecting at all. I run the risk of having outdated software, but that can be addressed by storing the software in a machine that's not connected to The Internet in any way, so it can't really do anything/talk to a C2 server (if someone does decide to execute an 0day with the software or inject malicious code via a rogue update).
- trynton 5y ago
  @cyberlab: “Is there the rare case that we shouldn't update because the update could contain a malicious payload?” You don't ever update your security “computers” from some third-party outsourcer. What you do is have your own people constantly probing their own systems for potential vulnerability and patching it themselves. * jeez ..SolarWinds run their stuff on FTP and "active directory". It's got to be a joke. * “computers” .. not allowed to use the 'W' word ;]
- deleted 5y ago
  [deleted]
- covidthrow 5y ago
  Plaintext transport doesn't matter if at least one part of the payload chain is cryptographically protected/verified. If you have a machine that's air-gapped and its only IO is strictly humans (read: keyboard/screen, not USB or other electronic means) then your weak point is the human, so center your security around that. You can look at security of lottery machines to get a good idea how that's handled. But if you're updating the machine with updates, then it doesn't really fit that criteria, soooo....
germinalphrase 5y ago
“A ‘Worst Nightmare’ cyberattack” that we all... just take in stride? Either the consequences are themselves clandestine, or cyberattacks aren’t as meaningful as our headlines would indicate.
- exporectomy 5y ago
  The worst nightmare of the vice president of security at SolarWinds, not of the average person.
  - zaphar 5y ago
    An attack directed at your own govt is potentially a nightmare for the average individual. If the wrong information is stolen it could be used much farther down the road. Your govt may find itself at a disadvantage at a critical moment. I'm a sense it's only not a nightmare if you aren't paying attention.
    - exporectomy 5y ago
      A worst nightmare for normal people would be something like a foreign country hacking all the US nuclear missiles and launching them against themselves. You know, an actual war caused by it, not just some minor intelligence advantage with no particular direct effect on normal people.
  - wglb 5y ago
    Unclear to me how worried he is. See https://www.turn-keytechnologies.com/blog/article/solarwinds-cybersecurity-made-it-vulnerable-to-an-attack/ https://www.turn-keytechnologies.com/blog/article/solarwinds... and other articles at https://ciexinc.com/blog/solarwinds-articles/culture.html https://ciexinc.com/blog/solarwinds-articles/culture.html.
  - aleister_777 5y ago
    At least 18k+ networks were compromised. I bet average people got caught in that net.
- 5y ago
1cvmask 5y ago
It’s nice how they equivocate over the ease of entry and their security policies: There was another unsettling report about passwords. A security researcher in Bangalore, India, named Vinoth Kumar told NPR that he had found the password to a server with SolarWinds apps and tools on a public message board and the password was: "solarwinds123." Kumar said he sent a message to SolarWinds in November and got an automated response back thanking him for his help and saying the problem had been fixed. When NPR asked SolarWinds' vice president of security, Brown, about this, he said that the password "had nothing to do with this event at all, it was a password to a FTP site." An FTP site is what you use to transfer files over the Internet. He said the password was shared by an intern and it was "not an account that was linked to our active directory."
- dogman144 5y ago
  How a Vp security can ignore a privesc risk like that is pretty inexcusable. Ever vuln falls on a risk mgmt spectrum but that’s a really nonsense answer to give. Weak PW mgmt on a FTP server that you let interns set should raise some areas of interest.
  - ridaj 5y ago
    I don't think they ignored it? Says it was addressed. I think it was OK to dispel the implication that the elaborate supply chain attack was allowed in the first place by sloppy pw practices. You don't want that to be the takeaway and then other companies thinking, well our pw management practice is really good so we don't have to worry about being a victim so much
    - rjzzleep 5y ago
      They did ignore it. I think it was over a year actually. I don't feel like looking it up right now but it was also discussed in another submission. Whatever the actual number was it was VERY long.
  - CyberRage 5y ago
    You don't get this kind of attack because you had an exposed FTP server. The attack implanted malicious code into their code, learning the tooling, process and responsibilities of the personal. They then reversed engineered the protocol and used it in their backdoor to look basically the same as regular communications. The issue is that we blindly trust 3rd party software that is used by hundreds of companies. this makes SolarWinds a prime target, one that is worth the efforts taken in this case.
trynton 5y ago
Don't put remote control software* on your security infrastructure such that when a third party gets hacked, your whole network is exposed. * The presence of such software making it that much more easy to hijack your “computers” * Please “Hacker News”, no more anti-Russian neocon propaganda. Who's really to blame is the idiot that put that configuration in, in the first place.
netfortius 5y ago
This article is a case of [a lot of] Monday morning quarterback[s]. Except for Mandia, I wouldn't allow any other exec(s) to speak about this, publicly, as to the why and how. Side note: I bet Bejtlich wishes he was still in that team ;-)
- wglb 5y ago
  The best set of articles in my opinion are these: https://ciexinc.com/blog/solarwinds-articles/zetter.html https://ciexinc.com/blog/solarwinds-articles/zetter.html
adolph 5y ago
How fortuitous is it that a months long investigation can be published right when the US announces sanctions? Great job National Radio! Like razor blades in peanut butter cups, says CrowdStrike.
- covidthrow 5y ago
  I fear that you're being downvoted for pointing out the even bigger threat of our national media's exceedingly more dangerous acts of propaganda, presumably because readers either don't recognize it or are wilfully blind to it because it reflects their own biases. The SolarWinds hack was a devastating attack on our sovereignty. And so is the other.
  - encryptluks2 5y ago
    The SolarWinds hack in addition to election interference should easily be seen as an attack worthy of taking out Putin IMO.
    - askl56 5y ago
      Yes, let's assassinate a world leader of the country with the most nukes in the world based on unfounded claims of election interference and a hack. Do you have any idea what America did in Russia in the immediate aftermath of the fall of the Soviet Union? The 1996 election in Russia which was majorly "interfered" with by Clinton: https://archive.is/R7i5u https://archive.is/R7i5u, not to mention the fact that most Russian hacking activities are done using NSA backdoors which were leaked? I'm surprised and disappointed to see such flagrant ivory tower imperialism on HN.
      - grosswait 5y ago
        Your cited article is interesting but does not support your assertion of major interference IMO. Policy maneuvers for political purposes are the norm for all countries right or wrong.
arminiusreturns 5y ago
Let me introduce you to PTECH and see if you still think Solarwinds was worse. Warning, a conspiracy rabbit hole lies this way, proceed with caution, lest your view of the world be challenged. A start: https://www.youtube.com/watch?v=UuZRMpt_Tas&t=195 https://www.youtube.com/watch?v=UuZRMpt_Tas&t=195
- tomComb 5y ago
  Given that audio and video are much less likely than written works to include references and are harder to reference themselves, I think they are poor evidence of anything contentions such as a conspiracy theory.
- matkoniecz 5y ago
  Is there something about that available in a serious form? (not a movie, especially Youtube movie)
  - arminiusreturns 5y ago
    edit: The topic level link in the youtube description sends you straight to the list of documentation... 1. http://en.wikipedia.org/wiki/Ptech http://en.wikipedia.org/wiki/Ptech 2. https://archive.org/details/GunsNButterIndiraSinghPtechAndThe911SoftwarePt1Pt2 https://archive.org/details/GunsNButterIndiraSinghPtechAndTh... 3. http://masshightech.bizjournals.com/masshightech/stories/2003/08/25/newscolumn2.html?t=printable http://masshightech.bizjournals.com/masshightech/stories/200... 4. http://archive.is/RuleH http://archive.is/RuleH 5. http://web.archive.org/web/20080905224929/http://www.theamericanmonitor.com/articles/albania_alqadi.html http://web.archive.org/web/20080905224929/http://www.theamer... 6. http://web.archive.org/web/20080820045652/http://www.total911.info/2005/03/wall-street-whistleblower-singh-on.html http://web.archive.org/web/20080820045652/http://www.total91... 7. http://web.archive.org/web/20080515202659/http://www.judicialwatch.org/printer_2469.shtml http://web.archive.org/web/20080515202659/http://www.judicia... 8. http://web.archive.org/web/20081015113454/http://911citizenswatch.org/?p=430 http://web.archive.org/web/20081015113454/http://911citizens... 9. http://web.archive.org/web/20080915185702/http://counterterrorismblog.org/2007/10/operation_greenquest_unresolve.php http://web.archive.org/web/20080915185702/http://counterterr... 10. http://rememberingmichael.wordpress.com/2008/03/20/in-memory-of-michael-corbin-who-kept-watch-held-the-light/#more-9 http://rememberingmichael.wordpress.com/2008/03/20/in-memory... 11. http://www.abovetopsecret.com/forum/thread183761/pg1 http://www.abovetopsecret.com/forum/thread183761/pg1
andix 5y ago
I‘m quite sure there are a lot of attacks like that. Most of them just never get noticed. The best backdoors are those, which are never found.
- makomk 5y ago
  Like, say, that backdoor someone wrote an article about recently which ran from RAM and had a sophisticated self-destruct mechanism that erased all traces if anyone tried to dump its memory? I wonder how many companies had exploits like that which they either didn't notice or didn't have the sophistication to actually catch and dump.
  - tkinom 5y ago
    There are defences for this: If one controls/monitors for every app in system for network access, as soon as any unusually network access are triggered, it is investigated and block. In my home windows setup, only windows defender, firefox and chrome are allowed out going internet access in regular base. Everything else are blocked. Windows update are only allowed when I in the mood for it (~once a year). Anyone can do this easily by control srvhost.exe 's internet access with windows firewall app.
    - sundarurfriend 5y ago
      > If one controls/monitors for every app in system Quis custodiet ipsos custodes? Who monitors the monitor? What happens when it is compromised, loopholed through, gets its inputs tampered with, etc.? For a home setup and its threat model, this sounds a simple, workable plan. When you're dealing with attacks of the level of sophistication described in the OP, trusting trust [1] becomes complicated and difficult. [1] http://users.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf http://users.ece.cmu.edu/~ganger/712.fall02/papers/p761-thom...
    - pitaj 5y ago
      Only pulling security updates once a year probably puts you at higher risk than you're protecting against by firewalling outbound traffic.
miohtama 5y ago
How much value SolarWinds shareholders have lost because of this? If it is not number one incentive for investors to fix, then there won’t be change in business practices. This is why GDPR in the EU has (some) teeth.
- wglb 5y ago
  There is a lot of legal action about this. It isn't clear that the stock is long term affected: https://ciexinc.com/blog/solarwinds-articles/stock.html https://ciexinc.com/blog/solarwinds-articles/stock.html
airstrike 5y ago
To me the "worst nightmare" was a story in the NYT about a hypothetical concerted attack against healthcare infrastructure, transit and more. Sadly I can't seem to find the link but it was a few years ago...
- vmh1928 5y ago
  Not so hypothetical. https://www.nytimes.com/2020/12/03/us/politics/vaccine-cyberattacks.html https://www.nytimes.com/2020/12/03/us/politics/vaccine-cyber... That's from December. Still going on. https://www.reuters.com/article/us-health-coronavirus-vaccines-cyber-idUSKBN2C12EU https://www.reuters.com/article/us-health-coronavirus-vaccin...
ridaj 5y ago
I agree with the parallels with aviation regulation, there needs to be something forcing a supplier's hand to solve this. The way to protect against supply chain attacks is to invest in a security-hardened build system (eg don't build releases on dev workstations, do them on build farms by build software that is the only thing able to access the release signing keys). This costs too much for most companies, so if they don't have the obligation to build it, they'll do features instead.
- Godel_unicode 5y ago
  > do them on build farms by build software that is the only thing able to access the release signing keys You're aware that this is exactly what solarwinds did, right?
  - ridaj 5y ago
    Well and then harden that obv
    - Godel_unicode 5y ago
      Which they did. The problem of hardening your build infrastructure against someone who has admin access for months is... non-trivial. This boils down to the question of should average companies be including the Russian intelligence services in their threat model? To paraphrase James Mickens great USENIX paper, if your threat model includes the SVR, you're going to be SVR'd upon.
      - Veserv 5y ago
        First, you do not need to be the Russian intelligence services to pull off this attack. Given prevailing trends in the vulnerabilities market this sort of attack would cost at most $1M to pull of which puts it within the capabilities of maybe ~50,000,000 individuals worldwide let alone organizations. If the SVR is anything like the CIA they are probably running at least 1,000 programs of similar scale simultaneously, so it is not as if the attack was supported by the full weight of the Russian intelligence services. Second, a company's threat model should include entities that want to attack them. Given that they are claiming the SVR wanted to and did attack them, it would be ridiculous to not include them since that would be empirical evidence that they are an actual threat actor. Even if we were to ignore empirical evidence any company like SolarWinds that sells to wide swaths of government agencies in critical capacities should absolutely be including foreign intelligence services in their threat models and should probably be required to demonstrate effectiveness against attacks funded to at least the $100M level since only at that level does it start to actually get problematic for state actors to run operations.
amaccuish 5y ago
> But as CrowdStrike's decryption program chewed its way through the zeroes and ones, Meyers' heart sank. The crime scene was a bust. It had been wiped down That's a lot of words to say, we don't know who did it. I had a quick look but couldn't find anything, why are the fingers being pointed at Russia?
- ipsin 5y ago
  Tool use, essentially. Of course that could be spoofed, but I think that's the origin of the claim. https://www.reuters.com/article/us-global-cyber-solarwinds/solarwinds-hackers-linked-to-known-russian-spying-tools-investigators-say-idUSKBN29G0XT https://www.reuters.com/article/us-global-cyber-solarwinds/s...
  - joe_the_user 5y ago
    Tools can be stolen. In fact, if we're claiming whoever did this was a super-genius, they would have stolen or spoofed the tools they used to point at someone else. Unless they were Russia being so clever they were pretending to be someone pretending to be Russia! Edit: Your link show Kaspersky labs making the claim that this was the FSB. Yet the West also claims Kaspersky is controlled by FSB! Well, you could say "they should know". Or maybe they want to humor the West so their ban will be lifted. Or maybe they aren't controlled by the FSB at all. But if the West can't figure that out, how do they expect to figure out the true origin of the hack. "A riddle wrapped in mystery, inside an enigma"
- wglb 5y ago
  There has been considerable coverage of this attribution: https://ciexinc.com/blog/solarwinds-articles/attribution.html https://ciexinc.com/blog/solarwinds-articles/attribution.htm...
  - amaccuish 5y ago
    The reply from ipsin was much more helpful. This list is mostly full of articles in general about the Solarwinds attack and others including "Supernova malware clues link Chinese threat group Spiral to SolarWinds server hacks (ZDNet)".
de6u99er 5y ago
Anyone know how the software update was actually compromised in the first place?
- lambdasquirrel 5y ago
  It’s kinda-sorta in the article. The hackers managed to put some hack into the Solarwinds build scripts. Nice writing I guess, but, what allowed them to get into the build scripts?
- matwood 5y ago
  The wiki page on the attack speculates an Office360 account was hacked. Presumably it was an account from an admin, and from there I could see them probing until finding credentials for the build system.
  - trashcan 5y ago
    No idea if it is related, but a SAML implementation security issue was disclosed the same (or very close) day that the SolarWinds attack became public knowledge. Maybe that gave them access to the admin account?
  - trulyme 5y ago
    So they hacked MS through SW which was hacked through MS? Ironic if true.
- wglb 5y ago
  This has been a puzzle for a while, and I am not yet satisfied with the answer so far. The closest to an explanation is found here: https://ciexinc.com/blog/solarwinds-articles/initial.html https://ciexinc.com/blog/solarwinds-articles/initial.html. I feel like this says "well, we are not really sure".
  - galaxyLogic 5y ago
    Two things, are they unsure the attack could have worked the way they describe, or are they unsure the attack actually happened that way? From the viewpoint of the public it would be important to know what made this attack possible, and how to defend against it, even if the actual attack was accomplished some other way. I guess what I'm asking is, do they know how to repeat this attack?
williesleg 5y ago
Yay china and russia!
seereadhack 5y ago
How might you compartmentalize admin access the whole way down the stack at the enterprise level? What if you were to start from scratch?
tptacek 5y ago
Network monitoring software is a key part of the backroom operations we never see. [...] By its very nature, it touches everything — which is why hacking it was genius. This is frustrating to read, since plenty of people did in fact warn that these kinds of systems were easy targets.
- bostonsre 5y ago
  Yea.. not sure I'd call it genius. Think if you ask anyone that is a little knowledgeable about what would be the juiciest target for a nation state to hack, a large portion of people would have said something like SolarWinds. It seems like SolarWinds should have known better themselves as well. There is no way that their upper management didn't know that they would be an amazing target for a hack. Supply chain attacks are not that new. Their lax security seems extremely negligent.
  - tptacek 5y ago
    Ninety-nine times out of a hundred, defenders call attacks "genius" as a way of subverting accountability. What makes this particular incident pernicious is that it already had a built-in deflection of accountability --- the responsibility for ensuring that SolarWinds was fit for purpose was diffuse; hundreds of giant companies with large security teams all believed it was someone else's job to verify that SolarWinds could safely deliver its functionality. I've worked with people who don't operate this way, and who take continuous flack from CIOs for spending resources on verification for COTS IT management tools. But those teams are, in my experience, very rare --- and the SolarWinds hack provides further evidence of that view. It's not a perfect predictor, but a reasonable rule of thumb: if you've never heard of a vendor's security team, chances are they barely have one. That's obviously true of... most vendors! So you should be careful when you select one for a role as sensitive as fleetwide agent-based monitoring, where a vulnerability or a software supply chain fuckup is going to create mass compromise. This seems so clear to me that it barely counts as insight.
    - comboy 5y ago
      Also their security team can just be a subgroup of coders who have some idea how their software executes. IMHO most sane vendors who want you to install something on your machine make it open source and use existing tools as much as possible. Doing it this way also decreases chances of some "temporary fix" changes on even otherwise secure software. Companies optimize for money, management tries to align with company values and engineers often just have to follow it. It's inevitable what trade-offs will be made unless there's some direct negative impact. For everybody selling their time and not being heavily invested, ignoring black swans and basically "eating tons of sugar" is the natural move.
sorokod 5y ago
"The tradecraft was phenomenal" Indeed, consider Figure 5 here [1]. A truly diabolical mastermind. But seriously, the article looks like window dressing for common incompetence. [1] https://www.microsoft.com/security/blog/2020/12/18/analyzing-solorigate-the-compromised-dll-file-that-started-a-sophisticated-cyberattack-and-how-microsoft-defender-helps-protect/ https://www.microsoft.com/security/blog/2020/12/18/analyzing...
- wglb 5y ago
  A common way to dodge accountability is to exaggerate the size of the enemy and his cleverness and goodness he had 1000 developers working on it. Nonetheless, there are some things that are kind of impressive. Inserting their own code into the build process without touching any file. But the real level of skill, I think, is the operational discipline exercised by the attackers. For example, waiting two weeks before doing anything, erasing traces of what they did, and targeting very specific sites.
  - Agingcoder 5y ago
    Security is not my line of work, but back in the days I spent a while exploring the field (I especially enjoyed reverse engineering / subverting software ). Why is this 'operational discipline' remarkable? I'd expect this to be common sense (including waiting for two weeks) , especially if I was a state actor and had spent a whole bunch of money on it? Or do you mean that the vast majority of hacking operations don't even bother to do this?
    - wglb 5y ago
      Contrast that with the smash and grab of the exchange vulnerability that followed. Yes, many operations make some kind of error that gives the whole thing away
- galaxyLogic 5y ago
  It seems to me to get the attack going the attackers needed to accomplish 2 things (at least): 1. Get the source-code of SolarWinds.Orion.Core.BusinessLayer.dll so they could know how to modify it 2. Get their modified source OR modified compiled DLL somehow into the build-pipeline at SolarWinds. If SolarWinds could have prevented either of these two things from happening this attack would not have worked. Am I correct?
cmclaughlin 5y ago
The article vaguely describes the build system being compromised. Have any details been published to indicate what build systems they were running and what the exploits were there?
- wglb 5y ago
  There is a lot of discussion about that: https://ciexinc.com/blog/solarwinds-articles/drilldown.html https://ciexinc.com/blog/solarwinds-articles/drilldown.html
dmcgee 5y ago
This is the line that got me: >And so we are fairly broadly deployed software and where we enjoy administrative privileges in customer environments. There is a lot of talk about shoring up security practices by many of the people quoted here. But something that would be hard to admit is that maybe they should not have administrative privileges in customer environments. Maybe they should not install agents on your machine. They would never recommend you to do so with anyone else, except them of course, because you can trust them.
- wglb 5y ago
  And you would think that a bit of analysis would be done on software and the company that built it for something that you install and give it full administrative control.
  - Craighead 5y ago
    But... then that would reveal how the software functions leading to possible copying, which cant be permitted
- thyrsus 5y ago
  I don't like the Solarwinds Linux agent. When I last looked, there was still an sh syntax error in the cron job it installed (look for a file named 1 in your root directory) and I couldn't reach anyone who could understand my bug report. It also frequently exhausted the space in its log partition. I replaced the agent with SNMP 3 read only access. I don't believe I'm authorized to describe my employer's current monitoring posture.
- galaxyLogic 5y ago
  > Maybe they should not install agents on your machine But, is it possible or practical to do network management without "agents" ?
wglb 5y ago
I've collected many articles about this--currently 2099 articles--covering lots of areas. The highest quality articles are at https://ciexinc.com/blog/solarwinds-articles/zetter.html https://ciexinc.com/blog/solarwinds-articles/zetter.html.
vzaliva 5y ago
I am curious about "compiler" attack they are mentioning. Looks like they compromized compiler used to build the code. Any more technical info on this aspect?
- georgiecasey 5y ago
  I think they compromised Teamcity, or at least I've heard a lot of mentions of Teamcity related to this hack, I could be completely wrong. But it's possible they just got admin access to the Teamcity build server and added their code just before the build was deployed. So not really compromising the compiler per se
bane 5y ago
I think the most important thing the SolarWinds hack has revealed is that the massive pile of paperwork that has to be filled out, full of security controls, to accredit system for government systems, is fairly useless. It's the digital equivalent of the Great Wall of China. Designed by bureaucrats, impressive in size, a massive effort, and ultimately not going to stop the Mongols anyways. Security paperwork is not security. More important I think is that the months and months it takes to usher things through the process forces things to be out of date which in itself creates security problems. An actual audit of the source code + running it in an instrumented live test environment to capture behavior is far better.
- asperous 5y ago
  They didn’t prevent this specific hack so they are completely useless?
  - DecoPerson 5y ago
    They didn’t say “completely useless,” they said “fairly useless.” If you look at this case even briefly, you should come to the conclusion that the “security paperwork” is fairly useless. An FTP server compromised because of a terrible password policy? No suspicious activity alerts of any kind? Executives who (based on their comments) are clearly ignorant of what makes software actually secure? What is the paperwork able to prevent, if it can’t prevent such fundamental problems?
    - bane 5y ago
      Exactly. It's wonderful that there's a few thousand controls for things like password length. But things that cause massive security nightmares like "does the CTO care about security?" or "has a pentest team reviewed and audited the source?" are so fundamental and should be triggered as any new system touches more and more systems. "Does this software touch literally every other system on the network?" should be a question that triggers a much more rigorous and deeply technical evaluation and review. But the current processes don't work that way, they purely paperwork drills that often demonstrably make systems less safe.
galaxyLogic 5y ago
From what I've read so far I haven't been able to gather how Solarwinds could have prevented this? In other words what were the critical failures in their defenses? Or is this kept non-public on purpose?
micelwell000011 5y ago
https://www.casanz.org.au/wp-content/uploads/fsqm-files/free-TikTok-Followers-tiktok-free-followers-0112.pdf https://www.casanz.org.au/wp-content/uploads/fsqm-files/free...

305 points by bokchoi 5y ago

> The routine update, it turns out, is no longer so routine Is there the rare case that we shouldn't update because the update could contain a malicious payload? If the update gets served over plaintext HTTP I would treat it as suspicious and may even block it from connecting at all. I run the risk of having outdated software, but that can be addressed by storing the software in a machine that's not connected to The Internet in any way, so it can't really do anything/talk to a C2 server (if someone does decide to execute an 0day with the software or inject malicious code via a rogue update).

trynton 5y ago
@cyberlab: “Is there the rare case that we shouldn't update because the update could contain a malicious payload?” You don't ever update your security “computers” from some third-party outsourcer. What you do is have your own people constantly probing their own systems for potential vulnerability and patching it themselves. * jeez ..SolarWinds run their stuff on FTP and "active directory". It's got to be a joke. * “computers” .. not allowed to use the 'W' word ;]
deleted 5y ago
[deleted]
covidthrow 5y ago
Plaintext transport doesn't matter if at least one part of the payload chain is cryptographically protected/verified. If you have a machine that's air-gapped and its only IO is strictly humans (read: keyboard/screen, not USB or other electronic means) then your weak point is the human, so center your security around that. You can look at security of lottery machines to get a good idea how that's handled. But if you're updating the machine with updates, then it doesn't really fit that criteria, soooo....

germinalphrase 5y ago

“A ‘Worst Nightmare’ cyberattack” that we all... just take in stride? Either the consequences are themselves clandestine, or cyberattacks aren’t as meaningful as our headlines would indicate.

exporectomy 5y ago
The worst nightmare of the vice president of security at SolarWinds, not of the average person.
- zaphar 5y ago
  An attack directed at your own govt is potentially a nightmare for the average individual. If the wrong information is stolen it could be used much farther down the road. Your govt may find itself at a disadvantage at a critical moment. I'm a sense it's only not a nightmare if you aren't paying attention.
  - exporectomy 5y ago
    A worst nightmare for normal people would be something like a foreign country hacking all the US nuclear missiles and launching them against themselves. You know, an actual war caused by it, not just some minor intelligence advantage with no particular direct effect on normal people.
- wglb 5y ago
  Unclear to me how worried he is. See https://www.turn-keytechnologies.com/blog/article/solarwinds-cybersecurity-made-it-vulnerable-to-an-attack/ https://www.turn-keytechnologies.com/blog/article/solarwinds... and other articles at https://ciexinc.com/blog/solarwinds-articles/culture.html https://ciexinc.com/blog/solarwinds-articles/culture.html.
- aleister_777 5y ago
  At least 18k+ networks were compromised. I bet average people got caught in that net.
5y ago

1cvmask 5y ago

It’s nice how they equivocate over the ease of entry and their security policies: There was another unsettling report about passwords. A security researcher in Bangalore, India, named Vinoth Kumar told NPR that he had found the password to a server with SolarWinds apps and tools on a public message board and the password was: "solarwinds123." Kumar said he sent a message to SolarWinds in November and got an automated response back thanking him for his help and saying the problem had been fixed. When NPR asked SolarWinds' vice president of security, Brown, about this, he said that the password "had nothing to do with this event at all, it was a password to a FTP site." An FTP site is what you use to transfer files over the Internet. He said the password was shared by an intern and it was "not an account that was linked to our active directory."

dogman144 5y ago
How a Vp security can ignore a privesc risk like that is pretty inexcusable. Ever vuln falls on a risk mgmt spectrum but that’s a really nonsense answer to give. Weak PW mgmt on a FTP server that you let interns set should raise some areas of interest.
- ridaj 5y ago
  I don't think they ignored it? Says it was addressed. I think it was OK to dispel the implication that the elaborate supply chain attack was allowed in the first place by sloppy pw practices. You don't want that to be the takeaway and then other companies thinking, well our pw management practice is really good so we don't have to worry about being a victim so much
  - rjzzleep 5y ago
    They did ignore it. I think it was over a year actually. I don't feel like looking it up right now but it was also discussed in another submission. Whatever the actual number was it was VERY long.
- CyberRage 5y ago
  You don't get this kind of attack because you had an exposed FTP server. The attack implanted malicious code into their code, learning the tooling, process and responsibilities of the personal. They then reversed engineered the protocol and used it in their backdoor to look basically the same as regular communications. The issue is that we blindly trust 3rd party software that is used by hundreds of companies. this makes SolarWinds a prime target, one that is worth the efforts taken in this case.

trynton 5y ago

Don't put remote control software* on your security infrastructure such that when a third party gets hacked, your whole network is exposed. * The presence of such software making it that much more easy to hijack your “computers” * Please “Hacker News”, no more anti-Russian neocon propaganda. Who's really to blame is the idiot that put that configuration in, in the first place.

netfortius 5y ago

This article is a case of [a lot of] Monday morning quarterback[s]. Except for Mandia, I wouldn't allow any other exec(s) to speak about this, publicly, as to the why and how. Side note: I bet Bejtlich wishes he was still in that team ;-)

wglb 5y ago
The best set of articles in my opinion are these: https://ciexinc.com/blog/solarwinds-articles/zetter.html https://ciexinc.com/blog/solarwinds-articles/zetter.html

adolph 5y ago

How fortuitous is it that a months long investigation can be published right when the US announces sanctions? Great job National Radio! Like razor blades in peanut butter cups, says CrowdStrike.

covidthrow 5y ago
I fear that you're being downvoted for pointing out the even bigger threat of our national media's exceedingly more dangerous acts of propaganda, presumably because readers either don't recognize it or are wilfully blind to it because it reflects their own biases. The SolarWinds hack was a devastating attack on our sovereignty. And so is the other.
- encryptluks2 5y ago
  The SolarWinds hack in addition to election interference should easily be seen as an attack worthy of taking out Putin IMO.
  - askl56 5y ago
    Yes, let's assassinate a world leader of the country with the most nukes in the world based on unfounded claims of election interference and a hack. Do you have any idea what America did in Russia in the immediate aftermath of the fall of the Soviet Union? The 1996 election in Russia which was majorly "interfered" with by Clinton: https://archive.is/R7i5u https://archive.is/R7i5u, not to mention the fact that most Russian hacking activities are done using NSA backdoors which were leaked? I'm surprised and disappointed to see such flagrant ivory tower imperialism on HN.
    - grosswait 5y ago
      Your cited article is interesting but does not support your assertion of major interference IMO. Policy maneuvers for political purposes are the norm for all countries right or wrong.

arminiusreturns 5y ago

Let me introduce you to PTECH and see if you still think Solarwinds was worse. Warning, a conspiracy rabbit hole lies this way, proceed with caution, lest your view of the world be challenged. A start: https://www.youtube.com/watch?v=UuZRMpt_Tas&t=195 https://www.youtube.com/watch?v=UuZRMpt_Tas&t=195

tomComb 5y ago
Given that audio and video are much less likely than written works to include references and are harder to reference themselves, I think they are poor evidence of anything contentions such as a conspiracy theory.
matkoniecz 5y ago
Is there something about that available in a serious form? (not a movie, especially Youtube movie)
- arminiusreturns 5y ago
  edit: The topic level link in the youtube description sends you straight to the list of documentation... 1. http://en.wikipedia.org/wiki/Ptech http://en.wikipedia.org/wiki/Ptech 2. https://archive.org/details/GunsNButterIndiraSinghPtechAndThe911SoftwarePt1Pt2 https://archive.org/details/GunsNButterIndiraSinghPtechAndTh... 3. http://masshightech.bizjournals.com/masshightech/stories/2003/08/25/newscolumn2.html?t=printable http://masshightech.bizjournals.com/masshightech/stories/200... 4. http://archive.is/RuleH http://archive.is/RuleH 5. http://web.archive.org/web/20080905224929/http://www.theamericanmonitor.com/articles/albania_alqadi.html http://web.archive.org/web/20080905224929/http://www.theamer... 6. http://web.archive.org/web/20080820045652/http://www.total911.info/2005/03/wall-street-whistleblower-singh-on.html http://web.archive.org/web/20080820045652/http://www.total91... 7. http://web.archive.org/web/20080515202659/http://www.judicialwatch.org/printer_2469.shtml http://web.archive.org/web/20080515202659/http://www.judicia... 8. http://web.archive.org/web/20081015113454/http://911citizenswatch.org/?p=430 http://web.archive.org/web/20081015113454/http://911citizens... 9. http://web.archive.org/web/20080915185702/http://counterterrorismblog.org/2007/10/operation_greenquest_unresolve.php http://web.archive.org/web/20080915185702/http://counterterr... 10. http://rememberingmichael.wordpress.com/2008/03/20/in-memory-of-michael-corbin-who-kept-watch-held-the-light/#more-9 http://rememberingmichael.wordpress.com/2008/03/20/in-memory... 11. http://www.abovetopsecret.com/forum/thread183761/pg1 http://www.abovetopsecret.com/forum/thread183761/pg1

andix 5y ago

I‘m quite sure there are a lot of attacks like that. Most of them just never get noticed. The best backdoors are those, which are never found.

makomk 5y ago
Like, say, that backdoor someone wrote an article about recently which ran from RAM and had a sophisticated self-destruct mechanism that erased all traces if anyone tried to dump its memory? I wonder how many companies had exploits like that which they either didn't notice or didn't have the sophistication to actually catch and dump.
- tkinom 5y ago
  There are defences for this: If one controls/monitors for every app in system for network access, as soon as any unusually network access are triggered, it is investigated and block. In my home windows setup, only windows defender, firefox and chrome are allowed out going internet access in regular base. Everything else are blocked. Windows update are only allowed when I in the mood for it (~once a year). Anyone can do this easily by control srvhost.exe 's internet access with windows firewall app.
  - sundarurfriend 5y ago
    > If one controls/monitors for every app in system Quis custodiet ipsos custodes? Who monitors the monitor? What happens when it is compromised, loopholed through, gets its inputs tampered with, etc.? For a home setup and its threat model, this sounds a simple, workable plan. When you're dealing with attacks of the level of sophistication described in the OP, trusting trust [1] becomes complicated and difficult. [1] http://users.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf http://users.ece.cmu.edu/~ganger/712.fall02/papers/p761-thom...
  - pitaj 5y ago
    Only pulling security updates once a year probably puts you at higher risk than you're protecting against by firewalling outbound traffic.

miohtama 5y ago

How much value SolarWinds shareholders have lost because of this? If it is not number one incentive for investors to fix, then there won’t be change in business practices. This is why GDPR in the EU has (some) teeth.

wglb 5y ago
There is a lot of legal action about this. It isn't clear that the stock is long term affected: https://ciexinc.com/blog/solarwinds-articles/stock.html https://ciexinc.com/blog/solarwinds-articles/stock.html

airstrike 5y ago

To me the "worst nightmare" was a story in the NYT about a hypothetical concerted attack against healthcare infrastructure, transit and more. Sadly I can't seem to find the link but it was a few years ago...

vmh1928 5y ago
Not so hypothetical. https://www.nytimes.com/2020/12/03/us/politics/vaccine-cyberattacks.html https://www.nytimes.com/2020/12/03/us/politics/vaccine-cyber... That's from December. Still going on. https://www.reuters.com/article/us-health-coronavirus-vaccines-cyber-idUSKBN2C12EU https://www.reuters.com/article/us-health-coronavirus-vaccin...

ridaj 5y ago

I agree with the parallels with aviation regulation, there needs to be something forcing a supplier's hand to solve this. The way to protect against supply chain attacks is to invest in a security-hardened build system (eg don't build releases on dev workstations, do them on build farms by build software that is the only thing able to access the release signing keys). This costs too much for most companies, so if they don't have the obligation to build it, they'll do features instead.

Godel_unicode 5y ago
> do them on build farms by build software that is the only thing able to access the release signing keys You're aware that this is exactly what solarwinds did, right?
- ridaj 5y ago
  Well and then harden that obv
  - Godel_unicode 5y ago
    Which they did. The problem of hardening your build infrastructure against someone who has admin access for months is... non-trivial. This boils down to the question of should average companies be including the Russian intelligence services in their threat model? To paraphrase James Mickens great USENIX paper, if your threat model includes the SVR, you're going to be SVR'd upon.
    - Veserv 5y ago
      First, you do not need to be the Russian intelligence services to pull off this attack. Given prevailing trends in the vulnerabilities market this sort of attack would cost at most $1M to pull of which puts it within the capabilities of maybe ~50,000,000 individuals worldwide let alone organizations. If the SVR is anything like the CIA they are probably running at least 1,000 programs of similar scale simultaneously, so it is not as if the attack was supported by the full weight of the Russian intelligence services. Second, a company's threat model should include entities that want to attack them. Given that they are claiming the SVR wanted to and did attack them, it would be ridiculous to not include them since that would be empirical evidence that they are an actual threat actor. Even if we were to ignore empirical evidence any company like SolarWinds that sells to wide swaths of government agencies in critical capacities should absolutely be including foreign intelligence services in their threat models and should probably be required to demonstrate effectiveness against attacks funded to at least the $100M level since only at that level does it start to actually get problematic for state actors to run operations.

amaccuish 5y ago

> But as CrowdStrike's decryption program chewed its way through the zeroes and ones, Meyers' heart sank. The crime scene was a bust. It had been wiped down That's a lot of words to say, we don't know who did it. I had a quick look but couldn't find anything, why are the fingers being pointed at Russia?

ipsin 5y ago
Tool use, essentially. Of course that could be spoofed, but I think that's the origin of the claim. https://www.reuters.com/article/us-global-cyber-solarwinds/solarwinds-hackers-linked-to-known-russian-spying-tools-investigators-say-idUSKBN29G0XT https://www.reuters.com/article/us-global-cyber-solarwinds/s...
- joe_the_user 5y ago
  Tools can be stolen. In fact, if we're claiming whoever did this was a super-genius, they would have stolen or spoofed the tools they used to point at someone else. Unless they were Russia being so clever they were pretending to be someone pretending to be Russia! Edit: Your link show Kaspersky labs making the claim that this was the FSB. Yet the West also claims Kaspersky is controlled by FSB! Well, you could say "they should know". Or maybe they want to humor the West so their ban will be lifted. Or maybe they aren't controlled by the FSB at all. But if the West can't figure that out, how do they expect to figure out the true origin of the hack. "A riddle wrapped in mystery, inside an enigma"
wglb 5y ago
There has been considerable coverage of this attribution: https://ciexinc.com/blog/solarwinds-articles/attribution.html https://ciexinc.com/blog/solarwinds-articles/attribution.htm...
- amaccuish 5y ago
  The reply from ipsin was much more helpful. This list is mostly full of articles in general about the Solarwinds attack and others including "Supernova malware clues link Chinese threat group Spiral to SolarWinds server hacks (ZDNet)".

de6u99er 5y ago

Anyone know how the software update was actually compromised in the first place?

lambdasquirrel 5y ago
It’s kinda-sorta in the article. The hackers managed to put some hack into the Solarwinds build scripts. Nice writing I guess, but, what allowed them to get into the build scripts?
matwood 5y ago
The wiki page on the attack speculates an Office360 account was hacked. Presumably it was an account from an admin, and from there I could see them probing until finding credentials for the build system.
- trashcan 5y ago
  No idea if it is related, but a SAML implementation security issue was disclosed the same (or very close) day that the SolarWinds attack became public knowledge. Maybe that gave them access to the admin account?
- trulyme 5y ago
  So they hacked MS through SW which was hacked through MS? Ironic if true.
wglb 5y ago
This has been a puzzle for a while, and I am not yet satisfied with the answer so far. The closest to an explanation is found here: https://ciexinc.com/blog/solarwinds-articles/initial.html https://ciexinc.com/blog/solarwinds-articles/initial.html. I feel like this says "well, we are not really sure".
- galaxyLogic 5y ago
  Two things, are they unsure the attack could have worked the way they describe, or are they unsure the attack actually happened that way? From the viewpoint of the public it would be important to know what made this attack possible, and how to defend against it, even if the actual attack was accomplished some other way. I guess what I'm asking is, do they know how to repeat this attack?

williesleg 5y ago

Yay china and russia!

seereadhack 5y ago

How might you compartmentalize admin access the whole way down the stack at the enterprise level? What if you were to start from scratch?

tptacek 5y ago

Network monitoring software is a key part of the backroom operations we never see. [...] By its very nature, it touches everything — which is why hacking it was genius. This is frustrating to read, since plenty of people did in fact warn that these kinds of systems were easy targets.

bostonsre 5y ago
Yea.. not sure I'd call it genius. Think if you ask anyone that is a little knowledgeable about what would be the juiciest target for a nation state to hack, a large portion of people would have said something like SolarWinds. It seems like SolarWinds should have known better themselves as well. There is no way that their upper management didn't know that they would be an amazing target for a hack. Supply chain attacks are not that new. Their lax security seems extremely negligent.
- tptacek 5y ago
  Ninety-nine times out of a hundred, defenders call attacks "genius" as a way of subverting accountability. What makes this particular incident pernicious is that it already had a built-in deflection of accountability --- the responsibility for ensuring that SolarWinds was fit for purpose was diffuse; hundreds of giant companies with large security teams all believed it was someone else's job to verify that SolarWinds could safely deliver its functionality. I've worked with people who don't operate this way, and who take continuous flack from CIOs for spending resources on verification for COTS IT management tools. But those teams are, in my experience, very rare --- and the SolarWinds hack provides further evidence of that view. It's not a perfect predictor, but a reasonable rule of thumb: if you've never heard of a vendor's security team, chances are they barely have one. That's obviously true of... most vendors! So you should be careful when you select one for a role as sensitive as fleetwide agent-based monitoring, where a vulnerability or a software supply chain fuckup is going to create mass compromise. This seems so clear to me that it barely counts as insight.
  - comboy 5y ago
    Also their security team can just be a subgroup of coders who have some idea how their software executes. IMHO most sane vendors who want you to install something on your machine make it open source and use existing tools as much as possible. Doing it this way also decreases chances of some "temporary fix" changes on even otherwise secure software. Companies optimize for money, management tries to align with company values and engineers often just have to follow it. It's inevitable what trade-offs will be made unless there's some direct negative impact. For everybody selling their time and not being heavily invested, ignoring black swans and basically "eating tons of sugar" is the natural move.

sorokod 5y ago

"The tradecraft was phenomenal" Indeed, consider Figure 5 here [1]. A truly diabolical mastermind. But seriously, the article looks like window dressing for common incompetence. [1] https://www.microsoft.com/security/blog/2020/12/18/analyzing-solorigate-the-compromised-dll-file-that-started-a-sophisticated-cyberattack-and-how-microsoft-defender-helps-protect/ https://www.microsoft.com/security/blog/2020/12/18/analyzing...

wglb 5y ago
A common way to dodge accountability is to exaggerate the size of the enemy and his cleverness and goodness he had 1000 developers working on it. Nonetheless, there are some things that are kind of impressive. Inserting their own code into the build process without touching any file. But the real level of skill, I think, is the operational discipline exercised by the attackers. For example, waiting two weeks before doing anything, erasing traces of what they did, and targeting very specific sites.
- Agingcoder 5y ago
  Security is not my line of work, but back in the days I spent a while exploring the field (I especially enjoyed reverse engineering / subverting software ). Why is this 'operational discipline' remarkable? I'd expect this to be common sense (including waiting for two weeks) , especially if I was a state actor and had spent a whole bunch of money on it? Or do you mean that the vast majority of hacking operations don't even bother to do this?
  - wglb 5y ago
    Contrast that with the smash and grab of the exchange vulnerability that followed. Yes, many operations make some kind of error that gives the whole thing away
galaxyLogic 5y ago
It seems to me to get the attack going the attackers needed to accomplish 2 things (at least): 1. Get the source-code of SolarWinds.Orion.Core.BusinessLayer.dll so they could know how to modify it 2. Get their modified source OR modified compiled DLL somehow into the build-pipeline at SolarWinds. If SolarWinds could have prevented either of these two things from happening this attack would not have worked. Am I correct?

cmclaughlin 5y ago

The article vaguely describes the build system being compromised. Have any details been published to indicate what build systems they were running and what the exploits were there?

wglb 5y ago
There is a lot of discussion about that: https://ciexinc.com/blog/solarwinds-articles/drilldown.html https://ciexinc.com/blog/solarwinds-articles/drilldown.html

dmcgee 5y ago

This is the line that got me: >And so we are fairly broadly deployed software and where we enjoy administrative privileges in customer environments. There is a lot of talk about shoring up security practices by many of the people quoted here. But something that would be hard to admit is that maybe they should not have administrative privileges in customer environments. Maybe they should not install agents on your machine. They would never recommend you to do so with anyone else, except them of course, because you can trust them.

wglb 5y ago
And you would think that a bit of analysis would be done on software and the company that built it for something that you install and give it full administrative control.
- Craighead 5y ago
  But... then that would reveal how the software functions leading to possible copying, which cant be permitted
thyrsus 5y ago
I don't like the Solarwinds Linux agent. When I last looked, there was still an sh syntax error in the cron job it installed (look for a file named 1 in your root directory) and I couldn't reach anyone who could understand my bug report. It also frequently exhausted the space in its log partition. I replaced the agent with SNMP 3 read only access. I don't believe I'm authorized to describe my employer's current monitoring posture.
galaxyLogic 5y ago
> Maybe they should not install agents on your machine But, is it possible or practical to do network management without "agents" ?

wglb 5y ago

I've collected many articles about this--currently 2099 articles--covering lots of areas. The highest quality articles are at https://ciexinc.com/blog/solarwinds-articles/zetter.html https://ciexinc.com/blog/solarwinds-articles/zetter.html.

vzaliva 5y ago

I am curious about "compiler" attack they are mentioning. Looks like they compromized compiler used to build the code. Any more technical info on this aspect?

georgiecasey 5y ago
I think they compromised Teamcity, or at least I've heard a lot of mentions of Teamcity related to this hack, I could be completely wrong. But it's possible they just got admin access to the Teamcity build server and added their code just before the build was deployed. So not really compromising the compiler per se

bane 5y ago

I think the most important thing the SolarWinds hack has revealed is that the massive pile of paperwork that has to be filled out, full of security controls, to accredit system for government systems, is fairly useless. It's the digital equivalent of the Great Wall of China. Designed by bureaucrats, impressive in size, a massive effort, and ultimately not going to stop the Mongols anyways. Security paperwork is not security. More important I think is that the months and months it takes to usher things through the process forces things to be out of date which in itself creates security problems. An actual audit of the source code + running it in an instrumented live test environment to capture behavior is far better.

asperous 5y ago
They didn’t prevent this specific hack so they are completely useless?
- DecoPerson 5y ago
  They didn’t say “completely useless,” they said “fairly useless.” If you look at this case even briefly, you should come to the conclusion that the “security paperwork” is fairly useless. An FTP server compromised because of a terrible password policy? No suspicious activity alerts of any kind? Executives who (based on their comments) are clearly ignorant of what makes software actually secure? What is the paperwork able to prevent, if it can’t prevent such fundamental problems?
  - bane 5y ago
    Exactly. It's wonderful that there's a few thousand controls for things like password length. But things that cause massive security nightmares like "does the CTO care about security?" or "has a pentest team reviewed and audited the source?" are so fundamental and should be triggered as any new system touches more and more systems. "Does this software touch literally every other system on the network?" should be a question that triggers a much more rigorous and deeply technical evaluation and review. But the current processes don't work that way, they purely paperwork drills that often demonstrably make systems less safe.

galaxyLogic 5y ago

From what I've read so far I haven't been able to gather how Solarwinds could have prevented this? In other words what were the critical failures in their defenses? Or is this kept non-public on purpose?

micelwell000011 5y ago

https://www.casanz.org.au/wp-content/uploads/fsqm-files/free-TikTok-Followers-tiktok-free-followers-0112.pdf https://www.casanz.org.au/wp-content/uploads/fsqm-files/free...