8 ms·

SolarWind, enough with the password already

219 points by troydavis 6y ago

troydavis 6y ago
tl;dr: thegrugq argues that it's possible for SolarWinds' security to have been inadequate, yet that security posture to have made no difference in whether they were hacked by the SVR. thegrugq argues it's very likely: > I’m perfectly willing to believe that their build servers were using “admin:admin” and that’s how the Russians gained access to inject their code… but, this was a clandestine intelligence operation. They did not succeed merely because SolarWind had poor password hygiene.
- dilyevsky 6y ago
  Nonsense argument. What’s the point of securing anything if THE KGB can cast a hack spell and get root on my server?
  - sydd 6y ago
    No, read the article. What can they do if someone from KGB will apply to a junior dev position and get the job? They will hand them all their keys.
    - tptacek 6y ago
      And there are teams, a very few of them large, where internal segmentation is strong enough to survive a compromised developer machine. But that's an extraordinarily high bar to clear and very few companies, even those with strong security teams, really manage to clear it.
    - dilyevsky 6y ago
      I’ve been hearing this argument for over 10 years now about nsa, idf, china and now the kgb. The truth is internal audits are quite effective at catching these. If nsa could just place a plant why did they spend so much effort to tap into companies’ inter-dc fibers?
      - ukj 6y ago
        It's not an XOR.
      - benlivengood 6y ago
        Because mass surveillance is useful to identify targets and moles are useful to dig up the rest of the information on targets.
- blincoln 6y ago
  I have mixed feelings about this. I agree with him that if the SVR specifically targeted a particular organization with a specific goal in mind, it wouldn't really matter if they had weak passwords or not. OTOH, I think it's not out of the question that one or more organizations (intelligence agencies, criminals, etc.) found the password and took advantage of it more as a "let's see where this thread leads" type of opportunistic attack, and all of the downstream consequences only happened because of that. I've never worked for an intelligence agency, but I've been a professional penetration tester for about a decade, and when I go after an organization, that's typically my approach: find the weakest links and start following them to see where they go. In a complex environment, usually that leads to control over everything sooner or later. Edit: just to clarify that last paragraph, what I'm getting at is that if I imagine myself in the shoes of an intelligence agency, the "organizations" I'd be going after would be foreign countries. I'm sure in some cases it would make sense to go after specific businesses of interest, but in the absence of legal restrictions, I'd be looking for the weakest links in entire industries that supported those countries in some way, not necessarily picking a specific business and targeting them. There are quite a few companies out there that make systems monitoring and administration software that would provide similar levels of access to a wide range of organizations if their build chains were compromised. The one that was compromised was the one that also had a publicly-exposed update server with a password that could have been obtained in at least two different ways.[1] Coincidence? Perhaps, but I don't think it's fair to just take it off the table. [1] Accidentally exposed in a public GitHub repo for many months, as well as being easily guessed. Either alone would have been enough. Both being true seems to me to make it more likely.
  - troydavis 6y ago
    I think that's a good summary. Also, it's easy to imagine the SVR poking at all companies which sell high-trust applications[1] to many government agencies, and running with the ones[2] that worked. Regarding whether the password thing was a coincidence, I wouldn't be surprised that, if other large enterprise software companies were severely hacked, similar stories surfaced. That doesn't mean it's a coincidence, of course, but may mean that this is average among enterprise software companies. One takeaway here is that software companies shipping trusted software to third-party networks have an exposure more like Google or the Federal Reserve, not like other software companies. That's not how (a lot of) the software industry has acted. [1]: NMS, systems management, facilities management, possibly CRM [2]: Where one was detected (because the attackers chose the wrong target in FireEye), others have probably occurred and/or are active now. While this was very sophisticated, it wasn't Stuxnet (https://blog.erratasec.com/2021/02/no-1000-engineers-were-not-needed-for.html https://blog.erratasec.com/2021/02/no-1000-engineers-were-no...).
- a1369209993 6y ago
  Not quite. Rather, they argue that it's theoretically possible (though not necessarily realistic) to have good enough security to resist targeted hacking by the KGB, but this requires vastly more than just better password hygiene.
  - Sebb767 6y ago
    > it's theoretically possible (though not necessarily realistic) to have good enough security to resist targeted hacking by the KGB It is possible to have a security level where the cost outweighs the benefit. If the KGB really wants to go all out, they could buy employees, burn zero days or even hold sysadmin families hostage - but that would be extremely expensive and risky and they'd really need a big reason to go that far. If your password is "admin" or "solarwinds123", on the other hand, the biggest expense is probably the employee time spent laughing in the coffee room. I agree that it is vastly expensive to have a security level high enough that the KGB realistically can not take over your network, but it's far cheaper and more realistic to have a security level where it's not worth the expense.
    - a1369209993 6y ago
      > I agree that it is vastly expensive to have a security level high enough that the KGB realistically can not take over your network, but it's far cheaper and more realistic to have a security level where it's not worth the expense. Yep. The former is possible in theory but not likely to happen. The latter is somewhat difficult, but ought to be table stakes for any company dealing with security-sensitive anything.
bob33212 6y ago
People want an easy fix. It is easier to blame a single person for now follow a single rule, than to set the standard that executives should put controls in place to ensure that policies are being followed.
- Judgmentality 6y ago
  I understand the spirit of your comment, but I want to point out > It is easier to blame a single person There is only one person to blame. The CEO.
  - crazygringo 6y ago
    Huh? But the board appointed the CEO. So why isn't it the board? Or wait, the board was appointed by shareholders. So why isn't it shareholders? And so on...
    - Judgmentality 6y ago
      The board can't be expected to know as much as the CEO, the shareholders can't be expected to know as much as the board, and so on. As far as I'm concerned, the #1 responsibility of the CEO is to take blame for fuckups. Yes, I consider that a higher priority than making profits. Because if the CEO is unable to make profits, then the CEO has to own the fuckup of not making profits.
      - crazygringo 6y ago
        But the CEO can't be expected to know as much as each VP, just like each VP doesn't know everything each manager knows, etc. Also, when profits aren't made, it's not the CEO who suffers. They already got their salary. Its shareholders who suffer. Sorry if it's not clear but my overall point is that accountability has to exist at all levels. The CEO isn't the position where all accountability emanates from or where it all stops. The CEO is held accountable to the board; VP's are held accountable to the CEO. The CEO is just one cog in the chain.
burnthrow 6y ago
People focus on the password because it's the only part of the story they can relate to or understand. Orange County Rep. Katie Porter: > "I've got a stronger password than 'solarwinds123' to stop my kids from watching too much YouTube on their iPad ... You and your company were supposed to be preventing the Russians from reading Defense Department emails!" Words fail.
- owenmarshall 6y ago
  Is she that wrong? I don’t think so. Do I think most private companies could defend against Double Dragon or Lazarus or Fancy Bear? No, if a state level adversary is attacking you and the payoff is that good, you are going to get popped. But a strong posture makes it harder, which means they throw more at you and you have a chance of picking up on the attack. Best case, anyways. Worst case, you get to testify to Congress that your security measures were top notch and industry leading. That sounds a shit ton better than “we left a screen door open and didn’t notice for months.”
  - burnthrow 6y ago
    She's wrong to imply that if only SolarWinds had followed her iPad password policy, the attack would have been stopped. And she's mistaken about Orion's use case, which has nothing to do with email security. And while Russia conducted this attack, I'm tired of the Russian scarecrow: SolarWinds' job here has nothing to do with Russia. But mostly I'm jaded by ambitious SoCal pols neglecting their districts to score easy points on national issues.
    - cobythedog 6y ago
      > She's wrong to imply that if only SolarWinds had followed her iPad password policy, the attack would have been stopped. I don't think she was implying that at all. She was highlighting that if they couldn't even do a basic thing like employing stronger, more complex passwords - how could they defend against Russians reading DoD emails.
  - throwawayboise 6y ago
tester756 6y ago
Of course, because hacking other countries in Russia is "legal" and they do it. The trade off is that cannot really go on good vacations with all that stolen money, because they might have a unexpected visit
OptionX 6y ago
Claming the hackers would have been able to compromise SolarWinds even with good security pratices does not absolve the company of having actual good security. I know there are people that can pick a lock. Still gonna have them on my doors anyway.
- tptacek 6y ago
  The article notes that none its analysis absolves SolarWinds, and repeatedly goes out of its way to knife them.
  - grugq 6y ago
    My problem is not with SolarWind but with the analysis that keeps raising the password as the one and only problem. “If only they had good password hygiene then the company would’ve been totally safe against the Russian intelligence services!” That is just not how things work. To go further. The password was on GitHub from 2017 to November 2019. The first test build to see if they could backdoor things was in October 2019. If the password was the problem, why wasn’t SolarWind hacked in 2017 or 18? The only explanation is that it wasn’t an operation that existed back then. It wasn’t a target for the SVR at that point in time, or they weren’t able to service it with their operational capacity. But regardless, the critical factor here is that the RIS started this operation, not that the password was bad or available on GitHub. (Or whatever the issue is with the password.) Let’s discuss whether the operational concept (CONOP) of hacking a civilian target to get into the supply chain and hit other targets is acceptable in cyber espionage. It seems to be acceptable because that is a methodology that everyone uses. My point has not been that you just can’t win against Ho Chi Minh, or that SolarWind was particularly negligent (or not, their security posture was abysmal but also irrelevant)... my point is that we should focus on what actually matters — the CONOP. Because if the US sanctions Russia for this operation then the US is locking itself and it’s allies into a position when this CONOP is off the table. If that is what everyone agrees with, fine. But it’s the real discussion to have. Not what sort of security SolarWind did or (realistically) did not have.
    - Dma54rhs 6y ago
wyldfire 6y ago
> There is no rule that would prohibit the SolarWinds espionage campaign which the US would be willing to abide by itself. Of course! > misunderstanding of the hack in the public sphere Hopefully that part is not misunderstood, right?
nominated1 6y ago
> Could SolarWind have been too difficult for the KGB to use them in an enablement operation? Yes, it is possible to achieve that level of security. Creating a strong fast detection capability with rapid remediation and incident response will make it hard for attackers to dwell for any length of time, or persist on the system after they gain access. It requires vigilance and some effort, but it can be done. Of course, SolarWind wasn’t close to reaching that level. Who is responsible for vetting these partners? What’s the process look like? Surely it’s more than “trust us, we gotz great securties”.
- swiley 6y ago
  "Everyone has been doing it this way, don't break stuff."
simias 6y ago
There's a lot of baseless conjecture in this as far as I can tell. >The SolarWind backdoor was deeply integrated into the code, it was injected during their build process, and there is no way that the server having a weak password was the pivotal factor. As if Russian Intelligence would just give up if there were a strong password instead! If this was a wikipedia article there'd be a [citation needed] every other word. I also wonder if they're not overplaying the skill of Russian cybersecurity agents. I'm sure they're good, I'm sure some of them are very, very good but the idea that basically they don't care about passwords is going too far IMO. The main advantage of being a state-sponsored hacker is that you have access to resources most other black hats couldn't dream of (like sending a team of burglars to ransack somebody's house, or physically threaten an employee) but that doesn't mean that they can stop obeying the laws of physics and algorithmics. >There is practically no chance that the server’s password was in anyway relevant to the hack overall. Source: my behind. I think the author has a good point that it's probably best to have a holistic approach about these hacks instead of focusing exclusively on some details, but the details do matter. After all, the big picture is nothing but a long series of details, isn't it? >Close does not count in security. In offensive security you’re either successful or not. When you’re dealing with access then the only possible states are: did it work? Yes or no. Whether you need 5 minutes or 5 weeks to get a shell, once you have that shell, it is the same level of game over. That’s what we’re talking about here. The technique used to gain access is a minor issue. Reductio ad absurdum. So what does the author want us to do then? Set our passwords to qwerty1234 and just give up? It's especially weird when the author a few paragraphs earlier states: >Could SolarWind have been too difficult for the KGB to use them in an enablement operation? Yes, it is possible to achieve that level of security. Creating a strong fast detection capability with rapid remediation and incident response will make it hard for attackers to dwell for any length of time, or persist on the system after they gain access. So it turns out that 5 minutes or 5 weeks does matter after all? I find very little of substance in this entire rant. Also the KGB doesn't exist anymore, I don't know if the author doesn't know that or decides to keep using it for stylistic reasons but real life is not an 80s American B movie.
- tptacek 6y ago
  It's not clear what you're really trying to argue here. Obviously, he's not suggesting we use "qwerty1234" as our passwords; it's you who's reducing arguments to absurdity here. I'm pretty sure Grugq is aware of the current names for the Russian IC agencies.
slim 6y ago
this guy makes a lot of unsubstantiated assumptions about how kgb works
- qbasic_forever 6y ago
  I would wager anyone with intimate knowledge of how the KGB works, and has the evidence to prove it probably isn't going to be writing publicly about it for very long...
  - closeparen 6y ago
    Forensics and attribution are big parts of the infosec world. Researchers study attacks, tools, payloads, etc. and get an idea of different threat actors, their levels of sophistication, and who might have done which one. I’m sure it’s never ironclad - the conclusion that it was a particular intelligence agency is just an educated guess, and sophisticated attackers might intentionally ape the signatures of others. But it’s not completely hopeless.
- mmaunder 6y ago
  Grugq is well known in the infosec industry and more experienced in the area than you realize. He's been writing about opsec for years, among other areas.
  - WarOnPrivacy 6y ago
    This is true. Grugq on a very short list of infosec people who have my implicit trust.
joncrane 6y ago
Keep in mind that there's a ton of institutional shorting on $SWI and that's where a lot of these attack articles are coming from. Having said that, SolarWinds is garbage software even without the security vulnerabilities and I hope it goes the way of the dodo. Source: I've had the misfortune of using it on multiple contracts.
- genmud 6y ago
  Because anyone who is smart or has competent security teams have been looking to drop them like a sack of potatoes. I work with a ton of customers that have/had SolarWinds and the general sentiment with them is that they are ripping it out and replacing with alternatives. I would be willing to bet that the majority of SolarWinds sales are relationship based, not based on technical wins. It is extremely dangerous for a company that is based on relationship sales to require their economic buyers spend a non-insignificant amount of political capital defending bad practices. TBH most buyers aren't going to stick their neck out for them unless they have a really good reason. Even then, security teams and legal teams might poo-poo the purchase and for most people in IT it is easier to find an alternative. Being reliant on relationship sales rather than technical wins is not by itself a bad sales or growth strategy... But as a company who has taken that approach, you have to ensure you don't do stupid stuff, or the stupid stuff isn't something that requires people to stick their neck out for you on. I imagine this is why institutional investors are shorting $SWI more so than anything. Their customer churn is going to probably not be pretty and they are going to have to work really hard and hope that in 2-3 quarters people have forgotten about it.
  - alfiedotwtf 6y ago
    FIFY: I would be willing to bet that the majority of Enterprise sales are relationship based, not based on technical wins.
  - joncrane 6y ago
    Their Q4 2020 Earnings were pretty good. They issued downward guidance on their Q1 2021 Earnings but it looks like they don't expect to lose THAT much business. Most of their subscriptions are yearly so most people are stuck paying them for the duration of the contract regardless of whether they ripped out their Solarwinds stack or not.
slt2021 6y ago
did we ever see post mortem from Solarwinds? How did attackers enter the network? Even if build server used admin:admin, this server was in the intranet. How did they get inside the network? we need post mortem and to understand the entire attack chain, rather than sit and speculate about the abilities of KGB/SVR. Truth is KGB/SVR employees are very dumb and routinely leave traces. Their best hackers are actually civilians with commercial interests who do black hat campaigns for them in exchange for cover/protection on russian soil, but are not officially employed/enlisted. I have a hard time believing these were russian state sponsored hackers, unless somebody provides the hard evidence
- gist 6y ago
  > did we ever see post mortem from Solarwinds? Other than to provide misinformation (to lead people astray) what is the advantage to solar winds to do a post mortem? Why educate people? Upside vs. downside? A businesses purpose is to act like a business. Not to educate (for lack of a better way to put it) the 'peanut' gallery, pundits, news outlets, bloggers or to improve security for others. Or to seem like 'a good company'. Nobody will deal with or not deal with Solar Winds based on what they say afterwords in a public and open forum. Privately and maybe under NDA sure but why broadcast this to everyone? (Answer is not 'well that's what you do').
  - samstave 6y ago
    A post-mortem allows for the company to perform an analysis of what exactly went wrong and what needs to be fixed etc... It also shows that they have a CSO role and they are trying to instill faith in their customers...
  - bscphil 6y ago
    Plenty of companies seem to think that seeming like "a good company" is a sufficient business reason to do a post-mortem. That's the thing about expectations: even if you can't come up with a rationale for doing something, other peoples' reactions to not doing the thing can be a sufficient justification for doing it. > well that's what you do I assume what this phrase is supposed to get at the sense that doing port mortems is the (morally) right thing to do and part of our duties as engineers to each other and to the public that has an interest in security. If that's the case then I have to disagree with you, that's an excellent reason to do it. (The fact that something would be the right thing to do means that you have a good reason to do the thing: namely, that it would be the right thing to do.) You can be cynical and say that "doing the right thing" is not going to be a good enough motivation to convince business X to do it, and that's fine, but it doesn't sound like that's what you're saying here.
jaredsohn 6y ago
I think I remember seeing solarwinds123 as a password around 2011 (perhaps as a default password within Orion?) but couldn't find a web search for it. But I did find this example which I find amusing: "For example if your account name is 'orion@mycompany.com' and the password is SolarWinds123, that's what you put in for the authentication." https://thwack.solarwinds.com/product-forums/network-performance-monitor-npm/f/forum/85223/setting-smtp-server-in-solarwinds https://thwack.solarwinds.com/product-forums/network-perform...
- jaredsohn 6y ago
  Did some more searching; Google's filter by date is broken since companies update old urls to include recent news headlines. https://www.google.com/search?q=solarwinds123&tbs=cdr:1,cd_min:1/1/2000,cd_max:1/1/2012 https://www.google.com/search?q=solarwinds123&tbs=cdr:1,cd_m...
mmaunder 6y ago
Grugq makes a few good points here: Sophisticated hacks employ a kill chain - think of it as what aviation calls a "cascade of failures". There's no single cause for the awful outcome, but instead a series of events where intercepting any of them could have mitigated the crash or in this case, the hack. For example, sure they got in, but they also remained undetected. If they didn't get in or if they were detected, the whole thing may have been mitigated. I also like how he's breaking away from just labeling the thing 'APT' and instead he describes the entity behind the attack, who they are, where they come from, what motivates them and how they are goal oriented rather than opportunistic. In other words, they didn't pick the target because of a weak password, they picked the target for strategic reasons. And finally the point of how well resourced and experienced these operatives are - or to use his phrase, they're pretty fucking metal. To unpack this a bit, the operatives targeting these kinds of attacks are well funded, experienced, patient, persistent, have large teams and once they've picked you, it's really hard to consider the odds stacked in your favor unless you truly understand what you're up against and have prepared accordingly.
- tmpz22 6y ago
  Reminds me of events like Pearl Harbor or even DDay where multiple signals existed to warn of the attacks and by some obnoxious sequence of ignorance they all went ignored.
  - mmaunder 6y ago
    That's perhaps a better analogy than my aviation one above. Exactly this. Break a link in the kill chain and it's prevented. So that's a slight advantage the defender has, but the trouble is most businesses (and individuals) have such a large attack surface that a determined attacker can usually find an alternative chain.
- grugq 6y ago
  Thanks
chokeartist 6y ago
Password blame aside, any CEO who blames an individual contributor just failed a major career test. They get paid the big bucks not only for the good times, but the bad times too. Total weasel move.
dj_mc_merlin 6y ago
The underestimation of the offense is done to ridiculous extents. The NSA pwned most of the world and exfiltrated data for years before it ever came to light (from one of their own!), yet all non-technical and even some technical people talk about security as if it's a bit you turn on or off. Unless you intend to spend as much money and resources on defense as USA/Russia/China does on offense, it's an uphill battle you will only seldom win. And you only have to lose once to lose almost everything.
- Sebb767 6y ago
  But that doesn't absolve you from trying! I totally agree with the point that a TLA has the resources to get into the network if they throw everything they have at you. A strong password might have changed nothing. But it is still a total failure on basic security on SolarWinds side. And honestly, the fact that they're now blaming an intern shows that this was not the one small weak point the attacker found, but a cultural problem.
walrus01 6y ago
What I find really funny about Solarwinds - is that in the medium to large ISP sector it's always been an absolute joke. Nobody of any consequence or real size has ever used it for network monitoring. Once you get to the scale of ISPs that have 50,000+ customers, or are supporting more than that through other smaller ISPs that are downstream of them - the monitoring and network automation tools are almost entirely open source, and some combination of GPL/LGPL/BSD/Apache/MIT license. Combined with custom things written in house to tie together different tools for a company's specific business needs. What you'll have typically is a collection of network equipment that may have closed-source operating systems (cisco, juniper routers and switches and similar, optical transport platforms from vendors like Infinera, Ciena), but everything managing and monitoring them is open source and runs on a *nix platform. If you have the in-house Linux/BSD knowledge to run the world's most powerful and popular open source networking tools, there is no need to ever touch solarwinds. My job interacts on a regular basis with all of the different pieces of the puzzle that make up solutions which are, in my opinion, vastly superior to Solarwinds. In the serious ISP business, if you ask the persons who admin the monitoring tools what they think of solarwinds, the answer you'll almost universally get back is "Windows GUI button pushing tools for enterprise end users who don't have the knowledge or motivation to really understand what's going on under the hood of their network".
- nikisweeting 6y ago
  Out of curiosity, can you give some examples of open-source monitoring tools that large-scale ISPs use?
  - walrus01 6y ago
    the first and most important thing is to have the correct network architecture and engineering to make effective use of the tools, and not have a network that needs a lot of babysitting in the first place. after that's taken care of as an over-arching and continual business process: there is no one single god box piece of software that is the be-all and do-all of network management/monitoring for an ISP. Some things come close, such as LibreNMS when used as the sole tool for a small ISP. But most often it is a patchwork quilt of many different things, each used for a discrete purpose. in no particular order: opennms a combination of (influxdb + telegraf + grafana) librenms provisioning and automation tools like ansible various in house things built on traditional RRA files and rrdtool tools like netbox for keeping track of datacenter customers/hosting environments phpipam or nipap for IP address management various self-hostable wiki software packages for internal documentation various types of self-hostable ticketing systems, monitoring systems that integrate with a customized asterisk system for NOC phone workflow 4 or 5 different tools that fill the same role as smokeping wireshark lots of different things for analyzing netflow data (Elastiflow or other) ELK stack stuff, elasticsearch/logstash/kibana, customized as needed. in house setups for openstreetmap tile servers and map presentation, to pull data from back-end mariadb databases and present them on monitoring displays. GIS software like QGIS and a PostGIS backend lots of different possible things done with custom code and postgresql, mysql/mariadb, or similar if you go through the PDF slideshows for the powerpoint decks at the last 4-5 years of the NANOG, RIPE and APNIC conferences you'll see discussion of some of the most popular network automation and monitoring tools.
Veserv 6y ago
The article has good points about the nature of attacks by a determined adversary. The concept of "The One Critical TTP" that companies tout to divert blame and that observers use to justify why they are not vulnerable to the same thing is utter nonsense. If somebody shoots a bullet at a bulletproof vest and it goes through you should not conclude that they just so happened to hit the only weak spot and if you just fix it everything is okay. You should instead assume that the bulletproof vest might have lots of problems at least against the gun you were using to test it. Successful breaches and attacks do not show you where your only weak points are, they show you the level of quality your process provides. To actually fix the problems the process, not the product, needs to be improved so that it is able to deliver higher quality outcomes. In the case of SolarWinds, we now know that the level of quality their process provides is insufficient to stop whoever attacked them. If we assume that it was a targeted attack by a nation-state actor, then we now know that they can not protect their customers against an actual adversary who had reason to attack them, willingness to attack them, and the ability to attack them. They are completely unable to defend against actual threats who will actually attack them. Lots of people will say: "Of course if a nation-state wants to attack me then there is nothing I can do, but why would they attack me?" Well, in this case, that is an actual threat. To provide an actual solution they do, in fact, actually need to be able to stop a nation-state. So, how does SolarWinds fare against a nation-state? They are not even on the same continent. Everybody thinks it is completely and utterly laughable that they would have had any hope of stopping them. Not just that, it is a forgone conclusion that if a nation-state wants to attack any commercial system they can with utter ease. It is not even viewed as a possibility for any currently deployed system to stop any nation-state from getting what they want. How far are these systems from stopping a nation-state? Well first we need to figure out what a nation-state can do. How valuable do you think the specs for the $1.5 trillion F-35 project would be to a peer adversary [1]? $100B? $10B? $1B? At the very least I would state that if a peer adversary could get the specs for the F-35 they would be willing to spend at least $1B on that project. So, to stop a nation-state you need a system that can protect against an attack funded to the $1B level. Assuming $500k/engineer-yr that is an attack with 2,000 engineer-years of development on it. There is no organization in the world who would even dare to claim that a team of 400 engineers working for 5 years could not completely and utterly compromise their systems. Even at 1/10th that nobody would dare to claim they could stop 40 engineers with 5 years. Even at 1/100th you would be hard pressed to actually find anybody who would claim they can stop 4 engineers for 5 years and you could probably count on one hand somebody who could actually deliver. The systems that are being deployed that are actually attacked by and must protect against nation-states need to improve by at least a factor of 100x before they can actually do their job. So, these systems are multiple orders of magnitudes away from achieving the minimum standard of functionality. What can be done about this state of affairs? Either we must do 100x better than the best deployed systems, or if that can not be done, then we must assume that these systems can be 100% guaranteed compromised and act accordingly. Either we must disconnect these systems since they can not be defended, or the benefits of their use must be greater than the worst-case outcome of failure. [1] https://www.idga.org/archived-content/news/pentagon-admits-f-35-data-theft-is-a-major-problem https://www.idga.org/archived-content/news/pentagon-admits-f...
qbasic_forever 6y ago
I do think a lot of folks are missing the main point that no matter what security theater is in place, a state actor with enough motivation is going to breach it. They're not going to send someone after the password protected parts, they're going to send your recruiter the most irresistible candidate--the perfect background, right out of your favorite school and with expertise in exactly your tech stack. You'll get pages and pages of glowing recommendations from people inside and around the industry. They'll ace your interview loops, be loved by all your engineers and managers, and they won't bat an eye at the lowball first offer you give them. They'll move up the corporate ranks with ease and be everyone's friend... and then the best security in the world doesn't matter one bit.
- slt2021 6y ago
  care to add examples? has there ever been a case when state actor sent a rogue employee and succeeded? because in serious organizations there are robust defenses even against potentially rogue employees
  - freeone3000 6y ago
    Stuxnet is the one that comes to mind first, where a Polish mole was able to be hired at an Iranian nuclear enrichment facility.
    - boogies 6y ago
      Could you cite that? Why would they need to infect flash drives all around the world if they had a mole directly inside the facility? Edit: grepping Wikipedia for “mole” it does appear that there was an Iranian one working with the Dutch government. https://en.wikipedia.org/wiki/Stuxnet#cite_ref-156 https://en.wikipedia.org/wiki/Stuxnet#cite_ref-156
  - gcampos 6y ago
    Not exactly what OP mentioned, but something like that did happen in the past: https://www.theguardian.com/technology/2010/jul/14/russian-spy-worked-for-microsoft https://www.theguardian.com/technology/2010/jul/14/russian-s...
afrcnc 6y ago
Thanks Reuters for propagating that bug bounty hunter's speculation. Cause this is where all of this goes back to.
karmakaze 6y ago
I accidentally clicked on a link in the article and the headline made me laugh. > Former SolarWinds CEO blames intern for 'solarwinds123' password leak
dang 6y ago
Related ongoing thread: SolarWinds CEO blames intern for password leak - https://news.ycombinator.com/item?id=26284782 https://news.ycombinator.com/item?id=26284782
WarOnPrivacy 6y ago
> There is no rule that would prohibit the SolarWinds espionage campaign which the US would be willing to abide by itself. Amen brother.
wglb 6y ago
Not only was it a bad idea to throw some unnamed intern under the bus, it reveals how broken the culture is at SolarWinds that they would even think that this is relevant. It smacks of delegating blame to someone no longer their, and not accepting responsibility to a very bad series of decisions.

219 points by troydavis 6y ago

tl;dr: thegrugq argues that it's possible for SolarWinds' security to have been inadequate, yet that security posture to have made no difference in whether they were hacked by the SVR. thegrugq argues it's very likely: > I’m perfectly willing to believe that their build servers were using “admin:admin” and that’s how the Russians gained access to inject their code… but, this was a clandestine intelligence operation. They did not succeed merely because SolarWind had poor password hygiene.

dilyevsky 6y ago
Nonsense argument. What’s the point of securing anything if THE KGB can cast a hack spell and get root on my server?
- sydd 6y ago
  No, read the article. What can they do if someone from KGB will apply to a junior dev position and get the job? They will hand them all their keys.
  - tptacek 6y ago
    And there are teams, a very few of them large, where internal segmentation is strong enough to survive a compromised developer machine. But that's an extraordinarily high bar to clear and very few companies, even those with strong security teams, really manage to clear it.
  - dilyevsky 6y ago
    I’ve been hearing this argument for over 10 years now about nsa, idf, china and now the kgb. The truth is internal audits are quite effective at catching these. If nsa could just place a plant why did they spend so much effort to tap into companies’ inter-dc fibers?
    - ukj 6y ago
      It's not an XOR.
    - benlivengood 6y ago
      Because mass surveillance is useful to identify targets and moles are useful to dig up the rest of the information on targets.
blincoln 6y ago
I have mixed feelings about this. I agree with him that if the SVR specifically targeted a particular organization with a specific goal in mind, it wouldn't really matter if they had weak passwords or not. OTOH, I think it's not out of the question that one or more organizations (intelligence agencies, criminals, etc.) found the password and took advantage of it more as a "let's see where this thread leads" type of opportunistic attack, and all of the downstream consequences only happened because of that. I've never worked for an intelligence agency, but I've been a professional penetration tester for about a decade, and when I go after an organization, that's typically my approach: find the weakest links and start following them to see where they go. In a complex environment, usually that leads to control over everything sooner or later. Edit: just to clarify that last paragraph, what I'm getting at is that if I imagine myself in the shoes of an intelligence agency, the "organizations" I'd be going after would be foreign countries. I'm sure in some cases it would make sense to go after specific businesses of interest, but in the absence of legal restrictions, I'd be looking for the weakest links in entire industries that supported those countries in some way, not necessarily picking a specific business and targeting them. There are quite a few companies out there that make systems monitoring and administration software that would provide similar levels of access to a wide range of organizations if their build chains were compromised. The one that was compromised was the one that also had a publicly-exposed update server with a password that could have been obtained in at least two different ways.[1] Coincidence? Perhaps, but I don't think it's fair to just take it off the table. [1] Accidentally exposed in a public GitHub repo for many months, as well as being easily guessed. Either alone would have been enough. Both being true seems to me to make it more likely.
- troydavis 6y ago
  I think that's a good summary. Also, it's easy to imagine the SVR poking at all companies which sell high-trust applications[1] to many government agencies, and running with the ones[2] that worked. Regarding whether the password thing was a coincidence, I wouldn't be surprised that, if other large enterprise software companies were severely hacked, similar stories surfaced. That doesn't mean it's a coincidence, of course, but may mean that this is average among enterprise software companies. One takeaway here is that software companies shipping trusted software to third-party networks have an exposure more like Google or the Federal Reserve, not like other software companies. That's not how (a lot of) the software industry has acted. [1]: NMS, systems management, facilities management, possibly CRM [2]: Where one was detected (because the attackers chose the wrong target in FireEye), others have probably occurred and/or are active now. While this was very sophisticated, it wasn't Stuxnet (https://blog.erratasec.com/2021/02/no-1000-engineers-were-not-needed-for.html https://blog.erratasec.com/2021/02/no-1000-engineers-were-no...).
a1369209993 6y ago
Not quite. Rather, they argue that it's theoretically possible (though not necessarily realistic) to have good enough security to resist targeted hacking by the KGB, but this requires vastly more than just better password hygiene.
- Sebb767 6y ago
  > it's theoretically possible (though not necessarily realistic) to have good enough security to resist targeted hacking by the KGB It is possible to have a security level where the cost outweighs the benefit. If the KGB really wants to go all out, they could buy employees, burn zero days or even hold sysadmin families hostage - but that would be extremely expensive and risky and they'd really need a big reason to go that far. If your password is "admin" or "solarwinds123", on the other hand, the biggest expense is probably the employee time spent laughing in the coffee room. I agree that it is vastly expensive to have a security level high enough that the KGB realistically can not take over your network, but it's far cheaper and more realistic to have a security level where it's not worth the expense.
  - a1369209993 6y ago
    > I agree that it is vastly expensive to have a security level high enough that the KGB realistically can not take over your network, but it's far cheaper and more realistic to have a security level where it's not worth the expense. Yep. The former is possible in theory but not likely to happen. The latter is somewhat difficult, but ought to be table stakes for any company dealing with security-sensitive anything.

bob33212 6y ago

People want an easy fix. It is easier to blame a single person for now follow a single rule, than to set the standard that executives should put controls in place to ensure that policies are being followed.

Judgmentality 6y ago
I understand the spirit of your comment, but I want to point out > It is easier to blame a single person There is only one person to blame. The CEO.
- crazygringo 6y ago
  Huh? But the board appointed the CEO. So why isn't it the board? Or wait, the board was appointed by shareholders. So why isn't it shareholders? And so on...
  - Judgmentality 6y ago
    The board can't be expected to know as much as the CEO, the shareholders can't be expected to know as much as the board, and so on. As far as I'm concerned, the #1 responsibility of the CEO is to take blame for fuckups. Yes, I consider that a higher priority than making profits. Because if the CEO is unable to make profits, then the CEO has to own the fuckup of not making profits.
    - crazygringo 6y ago
      But the CEO can't be expected to know as much as each VP, just like each VP doesn't know everything each manager knows, etc. Also, when profits aren't made, it's not the CEO who suffers. They already got their salary. Its shareholders who suffer. Sorry if it's not clear but my overall point is that accountability has to exist at all levels. The CEO isn't the position where all accountability emanates from or where it all stops. The CEO is held accountable to the board; VP's are held accountable to the CEO. The CEO is just one cog in the chain.

burnthrow 6y ago

People focus on the password because it's the only part of the story they can relate to or understand. Orange County Rep. Katie Porter: > "I've got a stronger password than 'solarwinds123' to stop my kids from watching too much YouTube on their iPad ... You and your company were supposed to be preventing the Russians from reading Defense Department emails!" Words fail.

owenmarshall 6y ago
Is she that wrong? I don’t think so. Do I think most private companies could defend against Double Dragon or Lazarus or Fancy Bear? No, if a state level adversary is attacking you and the payoff is that good, you are going to get popped. But a strong posture makes it harder, which means they throw more at you and you have a chance of picking up on the attack. Best case, anyways. Worst case, you get to testify to Congress that your security measures were top notch and industry leading. That sounds a shit ton better than “we left a screen door open and didn’t notice for months.”
- burnthrow 6y ago
  She's wrong to imply that if only SolarWinds had followed her iPad password policy, the attack would have been stopped. And she's mistaken about Orion's use case, which has nothing to do with email security. And while Russia conducted this attack, I'm tired of the Russian scarecrow: SolarWinds' job here has nothing to do with Russia. But mostly I'm jaded by ambitious SoCal pols neglecting their districts to score easy points on national issues.
  - cobythedog 6y ago
    > She's wrong to imply that if only SolarWinds had followed her iPad password policy, the attack would have been stopped. I don't think she was implying that at all. She was highlighting that if they couldn't even do a basic thing like employing stronger, more complex passwords - how could they defend against Russians reading DoD emails.
- throwawayboise 6y ago

tester756 6y ago

Of course, because hacking other countries in Russia is "legal" and they do it. The trade off is that cannot really go on good vacations with all that stolen money, because they might have a unexpected visit

OptionX 6y ago

Claming the hackers would have been able to compromise SolarWinds even with good security pratices does not absolve the company of having actual good security. I know there are people that can pick a lock. Still gonna have them on my doors anyway.

tptacek 6y ago
The article notes that none its analysis absolves SolarWinds, and repeatedly goes out of its way to knife them.
- grugq 6y ago
  My problem is not with SolarWind but with the analysis that keeps raising the password as the one and only problem. “If only they had good password hygiene then the company would’ve been totally safe against the Russian intelligence services!” That is just not how things work. To go further. The password was on GitHub from 2017 to November 2019. The first test build to see if they could backdoor things was in October 2019. If the password was the problem, why wasn’t SolarWind hacked in 2017 or 18? The only explanation is that it wasn’t an operation that existed back then. It wasn’t a target for the SVR at that point in time, or they weren’t able to service it with their operational capacity. But regardless, the critical factor here is that the RIS started this operation, not that the password was bad or available on GitHub. (Or whatever the issue is with the password.) Let’s discuss whether the operational concept (CONOP) of hacking a civilian target to get into the supply chain and hit other targets is acceptable in cyber espionage. It seems to be acceptable because that is a methodology that everyone uses. My point has not been that you just can’t win against Ho Chi Minh, or that SolarWind was particularly negligent (or not, their security posture was abysmal but also irrelevant)... my point is that we should focus on what actually matters — the CONOP. Because if the US sanctions Russia for this operation then the US is locking itself and it’s allies into a position when this CONOP is off the table. If that is what everyone agrees with, fine. But it’s the real discussion to have. Not what sort of security SolarWind did or (realistically) did not have.
  - Dma54rhs 6y ago

wyldfire 6y ago

> There is no rule that would prohibit the SolarWinds espionage campaign which the US would be willing to abide by itself. Of course! > misunderstanding of the hack in the public sphere Hopefully that part is not misunderstood, right?

nominated1 6y ago

> Could SolarWind have been too difficult for the KGB to use them in an enablement operation? Yes, it is possible to achieve that level of security. Creating a strong fast detection capability with rapid remediation and incident response will make it hard for attackers to dwell for any length of time, or persist on the system after they gain access. It requires vigilance and some effort, but it can be done. Of course, SolarWind wasn’t close to reaching that level. Who is responsible for vetting these partners? What’s the process look like? Surely it’s more than “trust us, we gotz great securties”.

swiley 6y ago
"Everyone has been doing it this way, don't break stuff."

simias 6y ago

There's a lot of baseless conjecture in this as far as I can tell. >The SolarWind backdoor was deeply integrated into the code, it was injected during their build process, and there is no way that the server having a weak password was the pivotal factor. As if Russian Intelligence would just give up if there were a strong password instead! If this was a wikipedia article there'd be a [citation needed] every other word. I also wonder if they're not overplaying the skill of Russian cybersecurity agents. I'm sure they're good, I'm sure some of them are very, very good but the idea that basically they don't care about passwords is going too far IMO. The main advantage of being a state-sponsored hacker is that you have access to resources most other black hats couldn't dream of (like sending a team of burglars to ransack somebody's house, or physically threaten an employee) but that doesn't mean that they can stop obeying the laws of physics and algorithmics. >There is practically no chance that the server’s password was in anyway relevant to the hack overall. Source: my behind. I think the author has a good point that it's probably best to have a holistic approach about these hacks instead of focusing exclusively on some details, but the details do matter. After all, the big picture is nothing but a long series of details, isn't it? >Close does not count in security. In offensive security you’re either successful or not. When you’re dealing with access then the only possible states are: did it work? Yes or no. Whether you need 5 minutes or 5 weeks to get a shell, once you have that shell, it is the same level of game over. That’s what we’re talking about here. The technique used to gain access is a minor issue. Reductio ad absurdum. So what does the author want us to do then? Set our passwords to qwerty1234 and just give up? It's especially weird when the author a few paragraphs earlier states: >Could SolarWind have been too difficult for the KGB to use them in an enablement operation? Yes, it is possible to achieve that level of security. Creating a strong fast detection capability with rapid remediation and incident response will make it hard for attackers to dwell for any length of time, or persist on the system after they gain access. So it turns out that 5 minutes or 5 weeks does matter after all? I find very little of substance in this entire rant. Also the KGB doesn't exist anymore, I don't know if the author doesn't know that or decides to keep using it for stylistic reasons but real life is not an 80s American B movie.

tptacek 6y ago
It's not clear what you're really trying to argue here. Obviously, he's not suggesting we use "qwerty1234" as our passwords; it's you who's reducing arguments to absurdity here. I'm pretty sure Grugq is aware of the current names for the Russian IC agencies.

slim 6y ago

this guy makes a lot of unsubstantiated assumptions about how kgb works

qbasic_forever 6y ago
I would wager anyone with intimate knowledge of how the KGB works, and has the evidence to prove it probably isn't going to be writing publicly about it for very long...
- closeparen 6y ago
  Forensics and attribution are big parts of the infosec world. Researchers study attacks, tools, payloads, etc. and get an idea of different threat actors, their levels of sophistication, and who might have done which one. I’m sure it’s never ironclad - the conclusion that it was a particular intelligence agency is just an educated guess, and sophisticated attackers might intentionally ape the signatures of others. But it’s not completely hopeless.
mmaunder 6y ago
Grugq is well known in the infosec industry and more experienced in the area than you realize. He's been writing about opsec for years, among other areas.
- WarOnPrivacy 6y ago
  This is true. Grugq on a very short list of infosec people who have my implicit trust.

joncrane 6y ago

Keep in mind that there's a ton of institutional shorting on $SWI and that's where a lot of these attack articles are coming from. Having said that, SolarWinds is garbage software even without the security vulnerabilities and I hope it goes the way of the dodo. Source: I've had the misfortune of using it on multiple contracts.

genmud 6y ago
Because anyone who is smart or has competent security teams have been looking to drop them like a sack of potatoes. I work with a ton of customers that have/had SolarWinds and the general sentiment with them is that they are ripping it out and replacing with alternatives. I would be willing to bet that the majority of SolarWinds sales are relationship based, not based on technical wins. It is extremely dangerous for a company that is based on relationship sales to require their economic buyers spend a non-insignificant amount of political capital defending bad practices. TBH most buyers aren't going to stick their neck out for them unless they have a really good reason. Even then, security teams and legal teams might poo-poo the purchase and for most people in IT it is easier to find an alternative. Being reliant on relationship sales rather than technical wins is not by itself a bad sales or growth strategy... But as a company who has taken that approach, you have to ensure you don't do stupid stuff, or the stupid stuff isn't something that requires people to stick their neck out for you on. I imagine this is why institutional investors are shorting $SWI more so than anything. Their customer churn is going to probably not be pretty and they are going to have to work really hard and hope that in 2-3 quarters people have forgotten about it.
- alfiedotwtf 6y ago
  FIFY: I would be willing to bet that the majority of Enterprise sales are relationship based, not based on technical wins.
- joncrane 6y ago
  Their Q4 2020 Earnings were pretty good. They issued downward guidance on their Q1 2021 Earnings but it looks like they don't expect to lose THAT much business. Most of their subscriptions are yearly so most people are stuck paying them for the duration of the contract regardless of whether they ripped out their Solarwinds stack or not.

slt2021 6y ago

did we ever see post mortem from Solarwinds? How did attackers enter the network? Even if build server used admin:admin, this server was in the intranet. How did they get inside the network? we need post mortem and to understand the entire attack chain, rather than sit and speculate about the abilities of KGB/SVR. Truth is KGB/SVR employees are very dumb and routinely leave traces. Their best hackers are actually civilians with commercial interests who do black hat campaigns for them in exchange for cover/protection on russian soil, but are not officially employed/enlisted. I have a hard time believing these were russian state sponsored hackers, unless somebody provides the hard evidence

gist 6y ago
> did we ever see post mortem from Solarwinds? Other than to provide misinformation (to lead people astray) what is the advantage to solar winds to do a post mortem? Why educate people? Upside vs. downside? A businesses purpose is to act like a business. Not to educate (for lack of a better way to put it) the 'peanut' gallery, pundits, news outlets, bloggers or to improve security for others. Or to seem like 'a good company'. Nobody will deal with or not deal with Solar Winds based on what they say afterwords in a public and open forum. Privately and maybe under NDA sure but why broadcast this to everyone? (Answer is not 'well that's what you do').
- samstave 6y ago
  A post-mortem allows for the company to perform an analysis of what exactly went wrong and what needs to be fixed etc... It also shows that they have a CSO role and they are trying to instill faith in their customers...
- bscphil 6y ago
  Plenty of companies seem to think that seeming like "a good company" is a sufficient business reason to do a post-mortem. That's the thing about expectations: even if you can't come up with a rationale for doing something, other peoples' reactions to not doing the thing can be a sufficient justification for doing it. > well that's what you do I assume what this phrase is supposed to get at the sense that doing port mortems is the (morally) right thing to do and part of our duties as engineers to each other and to the public that has an interest in security. If that's the case then I have to disagree with you, that's an excellent reason to do it. (The fact that something would be the right thing to do means that you have a good reason to do the thing: namely, that it would be the right thing to do.) You can be cynical and say that "doing the right thing" is not going to be a good enough motivation to convince business X to do it, and that's fine, but it doesn't sound like that's what you're saying here.

jaredsohn 6y ago

I think I remember seeing solarwinds123 as a password around 2011 (perhaps as a default password within Orion?) but couldn't find a web search for it. But I did find this example which I find amusing: "For example if your account name is 'orion@mycompany.com' and the password is SolarWinds123, that's what you put in for the authentication." https://thwack.solarwinds.com/product-forums/network-performance-monitor-npm/f/forum/85223/setting-smtp-server-in-solarwinds https://thwack.solarwinds.com/product-forums/network-perform...

jaredsohn 6y ago
Did some more searching; Google's filter by date is broken since companies update old urls to include recent news headlines. https://www.google.com/search?q=solarwinds123&tbs=cdr:1,cd_min:1/1/2000,cd_max:1/1/2012 https://www.google.com/search?q=solarwinds123&tbs=cdr:1,cd_m...

mmaunder 6y ago

Grugq makes a few good points here: Sophisticated hacks employ a kill chain - think of it as what aviation calls a "cascade of failures". There's no single cause for the awful outcome, but instead a series of events where intercepting any of them could have mitigated the crash or in this case, the hack. For example, sure they got in, but they also remained undetected. If they didn't get in or if they were detected, the whole thing may have been mitigated. I also like how he's breaking away from just labeling the thing 'APT' and instead he describes the entity behind the attack, who they are, where they come from, what motivates them and how they are goal oriented rather than opportunistic. In other words, they didn't pick the target because of a weak password, they picked the target for strategic reasons. And finally the point of how well resourced and experienced these operatives are - or to use his phrase, they're pretty fucking metal. To unpack this a bit, the operatives targeting these kinds of attacks are well funded, experienced, patient, persistent, have large teams and once they've picked you, it's really hard to consider the odds stacked in your favor unless you truly understand what you're up against and have prepared accordingly.

tmpz22 6y ago
Reminds me of events like Pearl Harbor or even DDay where multiple signals existed to warn of the attacks and by some obnoxious sequence of ignorance they all went ignored.
- mmaunder 6y ago
  That's perhaps a better analogy than my aviation one above. Exactly this. Break a link in the kill chain and it's prevented. So that's a slight advantage the defender has, but the trouble is most businesses (and individuals) have such a large attack surface that a determined attacker can usually find an alternative chain.
grugq 6y ago
Thanks

chokeartist 6y ago

Password blame aside, any CEO who blames an individual contributor just failed a major career test. They get paid the big bucks not only for the good times, but the bad times too. Total weasel move.

dj_mc_merlin 6y ago

The underestimation of the offense is done to ridiculous extents. The NSA pwned most of the world and exfiltrated data for years before it ever came to light (from one of their own!), yet all non-technical and even some technical people talk about security as if it's a bit you turn on or off. Unless you intend to spend as much money and resources on defense as USA/Russia/China does on offense, it's an uphill battle you will only seldom win. And you only have to lose once to lose almost everything.

Sebb767 6y ago
But that doesn't absolve you from trying! I totally agree with the point that a TLA has the resources to get into the network if they throw everything they have at you. A strong password might have changed nothing. But it is still a total failure on basic security on SolarWinds side. And honestly, the fact that they're now blaming an intern shows that this was not the one small weak point the attacker found, but a cultural problem.

walrus01 6y ago

What I find really funny about Solarwinds - is that in the medium to large ISP sector it's always been an absolute joke. Nobody of any consequence or real size has ever used it for network monitoring. Once you get to the scale of ISPs that have 50,000+ customers, or are supporting more than that through other smaller ISPs that are downstream of them - the monitoring and network automation tools are almost entirely open source, and some combination of GPL/LGPL/BSD/Apache/MIT license. Combined with custom things written in house to tie together different tools for a company's specific business needs. What you'll have typically is a collection of network equipment that may have closed-source operating systems (cisco, juniper routers and switches and similar, optical transport platforms from vendors like Infinera, Ciena), but everything managing and monitoring them is open source and runs on a *nix platform. If you have the in-house Linux/BSD knowledge to run the world's most powerful and popular open source networking tools, there is no need to ever touch solarwinds. My job interacts on a regular basis with all of the different pieces of the puzzle that make up solutions which are, in my opinion, vastly superior to Solarwinds. In the serious ISP business, if you ask the persons who admin the monitoring tools what they think of solarwinds, the answer you'll almost universally get back is "Windows GUI button pushing tools for enterprise end users who don't have the knowledge or motivation to really understand what's going on under the hood of their network".

nikisweeting 6y ago
Out of curiosity, can you give some examples of open-source monitoring tools that large-scale ISPs use?
- walrus01 6y ago
  the first and most important thing is to have the correct network architecture and engineering to make effective use of the tools, and not have a network that needs a lot of babysitting in the first place. after that's taken care of as an over-arching and continual business process: there is no one single god box piece of software that is the be-all and do-all of network management/monitoring for an ISP. Some things come close, such as LibreNMS when used as the sole tool for a small ISP. But most often it is a patchwork quilt of many different things, each used for a discrete purpose. in no particular order: opennms a combination of (influxdb + telegraf + grafana) librenms provisioning and automation tools like ansible various in house things built on traditional RRA files and rrdtool tools like netbox for keeping track of datacenter customers/hosting environments phpipam or nipap for IP address management various self-hostable wiki software packages for internal documentation various types of self-hostable ticketing systems, monitoring systems that integrate with a customized asterisk system for NOC phone workflow 4 or 5 different tools that fill the same role as smokeping wireshark lots of different things for analyzing netflow data (Elastiflow or other) ELK stack stuff, elasticsearch/logstash/kibana, customized as needed. in house setups for openstreetmap tile servers and map presentation, to pull data from back-end mariadb databases and present them on monitoring displays. GIS software like QGIS and a PostGIS backend lots of different possible things done with custom code and postgresql, mysql/mariadb, or similar if you go through the PDF slideshows for the powerpoint decks at the last 4-5 years of the NANOG, RIPE and APNIC conferences you'll see discussion of some of the most popular network automation and monitoring tools.

Veserv 6y ago

The article has good points about the nature of attacks by a determined adversary. The concept of "The One Critical TTP" that companies tout to divert blame and that observers use to justify why they are not vulnerable to the same thing is utter nonsense. If somebody shoots a bullet at a bulletproof vest and it goes through you should not conclude that they just so happened to hit the only weak spot and if you just fix it everything is okay. You should instead assume that the bulletproof vest might have lots of problems at least against the gun you were using to test it. Successful breaches and attacks do not show you where your only weak points are, they show you the level of quality your process provides. To actually fix the problems the process, not the product, needs to be improved so that it is able to deliver higher quality outcomes. In the case of SolarWinds, we now know that the level of quality their process provides is insufficient to stop whoever attacked them. If we assume that it was a targeted attack by a nation-state actor, then we now know that they can not protect their customers against an actual adversary who had reason to attack them, willingness to attack them, and the ability to attack them. They are completely unable to defend against actual threats who will actually attack them. Lots of people will say: "Of course if a nation-state wants to attack me then there is nothing I can do, but why would they attack me?" Well, in this case, that is an actual threat. To provide an actual solution they do, in fact, actually need to be able to stop a nation-state. So, how does SolarWinds fare against a nation-state? They are not even on the same continent. Everybody thinks it is completely and utterly laughable that they would have had any hope of stopping them. Not just that, it is a forgone conclusion that if a nation-state wants to attack any commercial system they can with utter ease. It is not even viewed as a possibility for any currently deployed system to stop any nation-state from getting what they want. How far are these systems from stopping a nation-state? Well first we need to figure out what a nation-state can do. How valuable do you think the specs for the $1.5 trillion F-35 project would be to a peer adversary [1]? $100B? $10B? $1B? At the very least I would state that if a peer adversary could get the specs for the F-35 they would be willing to spend at least $1B on that project. So, to stop a nation-state you need a system that can protect against an attack funded to the $1B level. Assuming $500k/engineer-yr that is an attack with 2,000 engineer-years of development on it. There is no organization in the world who would even dare to claim that a team of 400 engineers working for 5 years could not completely and utterly compromise their systems. Even at 1/10th that nobody would dare to claim they could stop 40 engineers with 5 years. Even at 1/100th you would be hard pressed to actually find anybody who would claim they can stop 4 engineers for 5 years and you could probably count on one hand somebody who could actually deliver. The systems that are being deployed that are actually attacked by and must protect against nation-states need to improve by at least a factor of 100x before they can actually do their job. So, these systems are multiple orders of magnitudes away from achieving the minimum standard of functionality. What can be done about this state of affairs? Either we must do 100x better than the best deployed systems, or if that can not be done, then we must assume that these systems can be 100% guaranteed compromised and act accordingly. Either we must disconnect these systems since they can not be defended, or the benefits of their use must be greater than the worst-case outcome of failure. [1] https://www.idga.org/archived-content/news/pentagon-admits-f-35-data-theft-is-a-major-problem https://www.idga.org/archived-content/news/pentagon-admits-f...

qbasic_forever 6y ago

I do think a lot of folks are missing the main point that no matter what security theater is in place, a state actor with enough motivation is going to breach it. They're not going to send someone after the password protected parts, they're going to send your recruiter the most irresistible candidate--the perfect background, right out of your favorite school and with expertise in exactly your tech stack. You'll get pages and pages of glowing recommendations from people inside and around the industry. They'll ace your interview loops, be loved by all your engineers and managers, and they won't bat an eye at the lowball first offer you give them. They'll move up the corporate ranks with ease and be everyone's friend... and then the best security in the world doesn't matter one bit.

slt2021 6y ago
care to add examples? has there ever been a case when state actor sent a rogue employee and succeeded? because in serious organizations there are robust defenses even against potentially rogue employees
- freeone3000 6y ago
  Stuxnet is the one that comes to mind first, where a Polish mole was able to be hired at an Iranian nuclear enrichment facility.
  - boogies 6y ago
    Could you cite that? Why would they need to infect flash drives all around the world if they had a mole directly inside the facility? Edit: grepping Wikipedia for “mole” it does appear that there was an Iranian one working with the Dutch government. https://en.wikipedia.org/wiki/Stuxnet#cite_ref-156 https://en.wikipedia.org/wiki/Stuxnet#cite_ref-156
- gcampos 6y ago
  Not exactly what OP mentioned, but something like that did happen in the past: https://www.theguardian.com/technology/2010/jul/14/russian-spy-worked-for-microsoft https://www.theguardian.com/technology/2010/jul/14/russian-s...

afrcnc 6y ago

Thanks Reuters for propagating that bug bounty hunter's speculation. Cause this is where all of this goes back to.

karmakaze 6y ago

I accidentally clicked on a link in the article and the headline made me laugh. > Former SolarWinds CEO blames intern for 'solarwinds123' password leak

dang 6y ago

Related ongoing thread: SolarWinds CEO blames intern for password leak - https://news.ycombinator.com/item?id=26284782 https://news.ycombinator.com/item?id=26284782

WarOnPrivacy 6y ago

> There is no rule that would prohibit the SolarWinds espionage campaign which the US would be willing to abide by itself. Amen brother.

wglb 6y ago

Not only was it a bad idea to throw some unnamed intern under the bus, it reveals how broken the culture is at SolarWinds that they would even think that this is relevant. It smacks of delegating blame to someone no longer their, and not accepting responsibility to a very bad series of decisions.