6 ms·

SolarWinds exposed FTP credentials in Public Github Repo: US Government Breach

142 points by hackerpain 6y ago

Metacelsus 6y ago
What's the best way to prevent accidental github leaks? I once accidentally leaked an API key (albeit for a relatively unimportant service, and I realized soon afterwards).
- solotronics 6y ago
  A good key rotation scheme and separate API keys for different purposes. Precommit automated checks for secrets.
  - john-tells-all 6y ago
    Absolutely! - a good, tested key rotation practice will reduce the impact of any accidental exposures. It'll also help tremendously when staff changes, and you can do things like "rotate all keys every quarter" - separate API keys for different services and environments means that even if your staging network-layer key is exposed, it can't affect your staging database (= data), nor your production database (= critical data). - I've used "search for AWS secrets" as a pre-commit and also CI hook with great luck. I've even accidentally added a secret myself :) but with the checks the error was immediately found. No problems!
- ChrisSD 6y ago
  I'm not sure there's a "best way" aside from being vigilant. * Don't blindly add everything to git. * Check staged files before commit. * Store secrets separately, outside of your git folder (e.g. use another folder, service or environment variables). * If it must be in your git folder, then make use of .gitignore and store keys, etc separately from code in files with a unique file name pattern.
- hackerpain 6y ago
  For basic live monitoring -- https://www.shhgit.com/ https://www.shhgit.com/ It's possible to monitor them while pushing code, and to have a monitoring system deployed to track them. I was thinking of open-sourcing our GitHubSniper tool, it basically has API regex from nearly a thousand SaaS and PaaS services and frequently posted sensitive strings, can be deployed via docker ;) The only thing that stops me from releasing it is I have consistently earned more in bounties than the incentives of actually releasing it publicly. But will eventually release it :)
  - mellosouls 6y ago
    That looks both cool and dodgy at the same time - making potential security breaches immediately available to anybody looking at it is rather different from offering it to people as a private security check service.
  - albanberg 6y ago
    sshgit: How does the notification work? Only on desktop? Nevermind...I found the docs. :-)
  - deleted 6y ago
    [deleted]
- beaconstudios 6y ago
  Don't include any credentials in your code, pull them in through AWS secrets, environment variables, or other runtime configuration.
- gostsamo 6y ago
  Code reviews. If it is in the diff, you have an issue, so make someone check your work. Otherwise, secrets as environment variables, do not write them down in files, and change them regularly.
- lock-free 6y ago
  Assume that secrets are going to be leaked or discovered and change them regularly. Everything else is a delay tactic, people mess up (or get tricked into messing up).
  - justin_oaks 6y ago
    You can change secrets regularly, but if you don't change them often enough then you can still be compromised because of it. If you change them too often then it's a maintenance problem. You might even take an outage because you have mismatched credentials (i.e. application or database had the database credentials updated but the other one didn't) Changing them regularly IS necessary. So if you're going to do it you should use a credential service like Hashicorp Vault.
- pwdisswordfish0 6y ago
  > What's the best way to prevent accidental github leaks? Don't use GitHub. Guaranteed to have 100% efficacy.
- acdha 6y ago
  Pre-commit.com has built in checks for AWS and SSH keys. I add this hook which checks for passwords, high-entropy strings, etc. and has tools for managing false positives: https://github.com/Yelp/detect-secrets https://github.com/Yelp/detect-secrets
- montjoy 6y ago
  Run your own git server behind a VPN. It still won’t prevent people from doing deliberately stupid things but will cut down on accidents. If you are super paranoid you air-gap everything.
- BeefWellington 6y ago
  Detect-Secrets, TruffleHog, or GitLeaks configured as a pre-commit hook in your local git repo to block you from committing them at all. Not perfect but gets you maybe 80-90% of the way.
- 0xbadcafebee 6y ago
  This blog post by Duo Labs explains everything you need to know. It describes the problem in detail, breaks it down, and explains mitigations. Everyone should read this. https://duo.com/labs/research/how-to-monitor-github-for-secrets https://duo.com/labs/research/how-to-monitor-github-for-secr... Here is their comparison of pre-commit hooks and other tools. It's current and comprehensive. https://github.com/duo-labs/secret-bridge/blob/master/TOOLS.md https://github.com/duo-labs/secret-bridge/blob/master/TOOLS....
- tialaramex 6y ago
  Don't write keys into code, even "test" code. Discipline yourself (and your team) to use dedicated key storage even on test systems, (and of course that key storage isn't "Duh, a github repo" but an appropriate technology for your platform) If you instinctively want to write shiny_api_key = 'SOME.KEY.BOSS.EMAILED.ME' Force yourself to instead take the extra few minutes to do # my-cloud-vendor key-store create shiny_api_key SOME.KEY.BOSS.EMAILED.ME and then write shiny_api_key = key_store_access('shiny_api_key') This way, when six hours later you check that code in, to GitHub, and go home for the weekend, you won't get a call from the boss ruining your Sunday lie-in, "WTF Did you upload our $5000 shiny API key to GitHub? You are fired". This habit also allows for correct separation as an organisation grows, when you're big enough you want it to be possible for somebody to write code against test systems with test API keys, and that same code to work as written in production, even though production API keys are not accessible to the coders. Using key stores to mediate makes this much more likely to actually work.
  - 0xbadcafebee 6y ago
    Don't know why you were downvoted, this is 100% what everyone should be doing. Don't even write shiny_api_key = "A-CLEARLY-FAKE-KEY". Don't write ANY secret AT ALL in your code, no matter what. Even for test frameworks. Just don't do it. Build in the secret retriving function from day one. Have it grab from an environment variable and it will be 100% seamless from development to production. Add more complexity/security if and when you need it. This can later be leveraged to allow for zero-downtime credential rotations for running services, which allows for seamless use of ephemeral credentials, which is what we should all be moving towards.
  - yowlingcat 6y ago
    This is the wrong answer. At a company I worked at, we had a well known saying: "Favor mechanisms over good intentions." As the saying went, if good intentions worked, we "wouldn't need mechanisms." The right answer here (as stated in sibling comments) is to install a mechanism into the build. A push that accidentally commits credentials should break the build. How it goes about doing that is up to you, but the specific way I've done that in the past is a git hook [1] specifically the pre-commit hook. It looks like there's a reasonably well-updated package by Yelp to help do this kind of thing [2]. [1] https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks [2] https://github.com/Yelp/detect-secrets https://github.com/Yelp/detect-secrets
- tetha 6y ago
  Hm, we have multiple layers of protection of different kinds there, since defense should always be in depth. First off, I'm pushing the principle that secrets should only be issued, created, stored and distributed by the secret management system, vault in our case. If you never see a secret, and only instruct the automation how to handle it, it is much harder to leak the secret. Ideally, in production, no human should ever need to see or access the secret. Second, you should have good rotation procedures. Something will eventually leak, or someone will leave on bad terms and you need to be able to handle that. This can be simple - an HA infra service can just reload the core credentials from vault and be done, or somewhat harder, e.g. if customers need and use those passwords. And finally, you should have entropy scanners like trufflehog or github scanners as linked in other tools - both in you precommit hooks (so you cannot commit the secret) as well as in your CI & repository host, so someone who disabled the security scanner "because it was annoying" still gets caught. And I guess finally, whiskey for that friday evening someone still manages to leak something important. Some terminations at work have not been fun.
- vangelis 6y ago
  Using an external config source helps. It's harder to accidentally check in 'secrets.json' than an API key in code.
ardy42 6y ago
Sophistication isn't necessarily mean technical wizardry. IMHO, it's pretty sophisticated to thoroughly look for easily-exploitable goofs, and extract maximum value from them. The big advantage nation states have is probably mainly staffing and funding, and why use a zero-day when a leaked password will do?
- porknubbins 6y ago
  I can see how this could be called sophisticated, but worry about language inflation because we still need something to describe technical feats that are govt/military level like Stuxnet.
  - ardy42 6y ago
    > I can see how this could be called sophisticated, but worry about language inflation because we still need something to describe technical feats that are govt/military level like Stuxnet. To start, I want to clear a potential confusion in my previous comment. By "technical wizardry" I meant "technical hacking wizardry," e.g. buffer overflows, ROP, etc. That said, I think Stuxnet and this hack may actually both be characterized by some of the same kind of sophistication I was referring to. IIRC, one of the biggest unique things about Stuxnet was the application of professional software engineering practices to malware. It wasn't a quick and dirty thing that one or two guys whipped up, it was a project, developed professionally over time by a team, to meet a set of requirements and allow code-reuse, etc. Not so much hacking wizardry but an application of government-scale money and personnel.
- dx87 6y ago
  Yeah, sophistication to me sounds more like the skill of the hacker. For example, logging in with leaked creds but cleaning up all relevant logs, and doing it during normal business hours for that account is a more sophisticated attack than using a zero day remote exploit and leaving a mess of evidence on the target systems.
- deleted 6y ago
  [deleted]
- 6y ago
wolco2 6y ago
Github is a single point of failure for the open source community
- mistrial9 6y ago
  > a single point of failure for the open source community some actors who benefit from scale are purposefully trying to make Github fit that role, to gather, search and mine the work of others without paying or giving back.. but due to the distributed nature of git itself, there are alternatives
- creato 6y ago
  What does this have to do with this incident? Do you think attackers having to scan 10 or more open source repo hosts for leaks is impractical? If so, maybe that's a worse situation? After all, if it's harder for attackers to scan, it must be harder for"white hats" looking for leaks to report? If anything, this might make companies increase the risk they associate with open sourcing their code. This wouldn't have happened if solar winds were not uploading anything at all to github...
  - _jal 6y ago
    It isn't open sourcing that increases risk, it is the open repositories. It is a vital distinction to make. Code needs to be scanned for secrets even in closed repositories because of insider threats. But opening them up guarantees your mistakes will be caught by, ah, volunteer sysadmins. I understand you're pointing out that fuzzing the distinction is a way for someone with an anti-open-source agenda to advance their efforts, just trying to make the point explicit.
Nightshaxx 6y ago
Someone on HN called it before this broke. Governments are quick to blame "state-sponsored actors" to cover up for their own flawed security practices.
- AndrewBissell 6y ago
  The Washington Post immediately using the incidents to call for pushback against "Russia's secret war" doesn't inspire confidence in the official story either.
protocol_7 6y ago
Good time run to https://github.com/zricethezav/gitleaks https://github.com/zricethezav/gitleaks on your projects
danso 6y ago
I still think about how Instagram's entire kingdom (source code, SSL certs, AWS credentials) was left wide open because someone deployed an open source admin tool on a single server and never changed the default hardcoded secret key: https://news.ycombinator.com/item?id=10754194 https://news.ycombinator.com/item?id=10754194 https://web.archive.org/web/20151217205414/http://www.exfiltrated.com/research-Instagram-RCE.php https://web.archive.org/web/20151217205414/http://www.exfilt...
- ProAm 6y ago
  It's okay that person had to solve really tough programming interview questions on a whiteboard to get that gig.
  - jesselawson 6y ago
    ^^ Most underrated comment here lol
    - WrtCdEvrydy 6y ago
      Sadly "The Whiteboard" doesn't test for having a pulse, just memorization :P
      - notReally01 6y ago
        Biologically speaking it’s all memorization. You memorized how to obfuscate comprehensions in a thousand ways. High 5 Discovery of net new abstraction is more interesting than re-proving you know what a mutex or cloud stack look like.
  - deleted 6y ago
    [deleted]
  - jennyyang 6y ago
    2-3 whiteboard interviews with 2 questions each. All questions need to be answered correctly.
tyho 6y ago
When reading press releases about cyberattacks, substitute "embarrassing" for "sophisticated" to improve the accuracy of the release.
- mellosouls 6y ago
  See also "nation state".
  - phkahler 6y ago
    And substitute "teenager in his moms basement"?
    - mellosouls 6y ago
      That sounds about right.
    - erdos4d 6y ago
      Maybe. I'd bet a number of attacks attributed to nation states are actually just talented amateurs. There's a lot of smart people out there.
- ChrisKnott 6y ago
  The sophistication was in reference to the malware which made use of SolarWind's certificate, wasn't it? I don't see the need for this editorialised submission title.
bg24 6y ago
Sophistication is how the hackers hide their tracks and exfiltrate the stuff after they have come in. Coming in can be a simple stuff, or can be a zero day vulnerability.
SoylentOrange 6y ago
Are the authors suggesting the weak credentials were the only part of the attack? Or are they suggesting they were the first part in a rather sophisticated supply chain attack? The credentials by themselves don’t give execute privileges, or persistence, or in fact any access to non-SolarWinds systems. So both can be true.
- wil421 6y ago
  The author is just rehashing old events and making baseless speculations. Pure blog spam.
rahimiali 6y ago
it just looks like the upload credentials for an incoming directory were easy, but that doesn't imply a weakness. I can "upload" stuff to Google Drive. that doesn't mean Google is easy to hack.
- rurban 6y ago
  Not just the incoming ftp dir, the whole server was exposed, in and out. Plus the admin name and password
lock-free 6y ago
The article/twitter thread don't offer anything definitive. It could be this leaked password was how their supply chain got compromised, it might not be. Just anecdotally I'm looking at my own release infrastructure and it would take a bit more work to get compromised binaries to our customers than a leaked secret in a public git repo - and if it was reported that we leaked some key or password that gave the keys of the kingdom away, it would be trivial to change it and verify that we didn't push malware. This article can explain how malware made it into a download server, maybe. But does it explain the fact the malware was signed? Maybe, but that suggests a bigger fuck up than leaking a password.
- hackerpain 6y ago
  There was a case where we were able to gain access to Jira instance belonging to a bug bounty target (not even a Red Team engagement), and the Confluence wiki had every credential the company ever used. They used it as a secure credential storage. Majority of their internal panels didn't have a 2FA and we found internal VPN server credentials too (which didn't have source ip restrictions), so we could freely access tons of data. The bounty we got was around $1000 if I could remember, I think this would motivate a lot of hackers to not go the responsible disclosure way, as the incentives are ridiculously low. But I could access literally everything related to that organization and their clients from that single Github leak (of Jira creds).
  - lock-free 6y ago
    Kind of an example of a "bigger fuck up" I was alluding to
lovecg 6y ago
This is just pure speculation at this point. The facts are that a leak existed and allegedly was fixed after the report. It’s quite a jump to conclude that this exact leak was used in the attack.
latchkey 6y ago
> For the readers who couldn't fully understand the tweet, Vinoth had apparently gotten access to a SolarWinds FTP server on 19th November, 2020 which is more than 1 year ago. 2019.
qw3rty01 6y ago
The email mentioned in the tweet is over a year old, with the response fixing the issue over a year old...the tweet in question is pointing out that their decisions relating to security are extremely questionable, not that this was the way they got in. This is also what he said in the Q&A session: > I think the attackers must have used the same approach as the FTP server was open & credentials were not strong enough. But it was a sophisticated attack as the binaries were signed.
julienchastang 6y ago
FTP? FTP does not have encryption (in contrast to HTTPS), correct?
- notRobot 6y ago
  There are multiple ways to secure FTP: FTPS and SFTP. FTPS uses SSL. SFTP is technically a different protocol, and uses SSH.
  - julienchastang 6y ago
    From the screenshot, it appears they are using vanilla FTP?
lee990 6y ago
This article is literally pure-speculative garbage. You have one guy saying he found an FTP credential in github and on the other hand a nation-state attack and putting two and two together with no proof or reasoning other than they both happened at some point in time.
perennate 6y ago
The article title is "SolarWinds exposed FTP credentials in Public Github Repo" (in 2019) which is far less definitive than the misleading title submitted on HN.
jpdb 6y ago
This article is extremely dishonest and should be removed. It offers zero proof this is what lead to the compromise. The leaked credentials were reported over a year ago and patched according to FireEye. There are already comments here from users who presumably haven't read the article and are parroting this as fact.
- tppiotrowski 6y ago
  Agree. Also: "A malicious software class was included among many other legitimate classes and then signed with a legitimate certificate," the Windows maker said. The resulting binary included a backdoor and was then discreetly distributed into targeted organizations." - which means that somehow a signing certificate must have been stolen or reverse engineered. Not simple.
- hackerpain 6y ago
  The article has mentioned its purely based on speculation, it doesn't claim - it was the cause. There has to be a bigger chain of attacks that led to the breach.
DobbsMT 6y ago
Given the timing of the NSA announcement that Russian intelligence was actively exploiting the VMware-related CVE-2020-4006 just a couple of weeks before the Fireeye and .Gov announcements, I'm wondering if that was the source of the intrusion. The platform in question handles mobile device management, identity brokering, and mobile email management, among other things. It would certainly align with the reports about Treasury and other departments having email compromised. https://www.securityweek.com/russian-hackers-exploiting-recently-patched-vmware-flaw-nsa-warns https://www.securityweek.com/russian-hackers-exploiting-rece... “The exploitation via command injection led to installation of a web shell and follow-on malicious activity where credentials in the form of SAML authentication assertions were generated and sent to Microsoft Active Directory Federation Services (ADFS), which in turn granted the actors access to protected data,”
dfpojsdfpojonsp 6y ago
This reads like it was written by a bot. It's filled with sentence fragments and grammar errors.

142 points by hackerpain 6y ago

What's the best way to prevent accidental github leaks? I once accidentally leaked an API key (albeit for a relatively unimportant service, and I realized soon afterwards).

solotronics 6y ago
A good key rotation scheme and separate API keys for different purposes. Precommit automated checks for secrets.
- john-tells-all 6y ago
  Absolutely! - a good, tested key rotation practice will reduce the impact of any accidental exposures. It'll also help tremendously when staff changes, and you can do things like "rotate all keys every quarter" - separate API keys for different services and environments means that even if your staging network-layer key is exposed, it can't affect your staging database (= data), nor your production database (= critical data). - I've used "search for AWS secrets" as a pre-commit and also CI hook with great luck. I've even accidentally added a secret myself :) but with the checks the error was immediately found. No problems!
ChrisSD 6y ago
I'm not sure there's a "best way" aside from being vigilant. * Don't blindly add everything to git. * Check staged files before commit. * Store secrets separately, outside of your git folder (e.g. use another folder, service or environment variables). * If it must be in your git folder, then make use of .gitignore and store keys, etc separately from code in files with a unique file name pattern.
hackerpain 6y ago
For basic live monitoring -- https://www.shhgit.com/ https://www.shhgit.com/ It's possible to monitor them while pushing code, and to have a monitoring system deployed to track them. I was thinking of open-sourcing our GitHubSniper tool, it basically has API regex from nearly a thousand SaaS and PaaS services and frequently posted sensitive strings, can be deployed via docker ;) The only thing that stops me from releasing it is I have consistently earned more in bounties than the incentives of actually releasing it publicly. But will eventually release it :)
- mellosouls 6y ago
  That looks both cool and dodgy at the same time - making potential security breaches immediately available to anybody looking at it is rather different from offering it to people as a private security check service.
- albanberg 6y ago
  sshgit: How does the notification work? Only on desktop? Nevermind...I found the docs. :-)
- deleted 6y ago
  [deleted]
beaconstudios 6y ago
Don't include any credentials in your code, pull them in through AWS secrets, environment variables, or other runtime configuration.
gostsamo 6y ago
Code reviews. If it is in the diff, you have an issue, so make someone check your work. Otherwise, secrets as environment variables, do not write them down in files, and change them regularly.
lock-free 6y ago
Assume that secrets are going to be leaked or discovered and change them regularly. Everything else is a delay tactic, people mess up (or get tricked into messing up).
- justin_oaks 6y ago
  You can change secrets regularly, but if you don't change them often enough then you can still be compromised because of it. If you change them too often then it's a maintenance problem. You might even take an outage because you have mismatched credentials (i.e. application or database had the database credentials updated but the other one didn't) Changing them regularly IS necessary. So if you're going to do it you should use a credential service like Hashicorp Vault.
pwdisswordfish0 6y ago
> What's the best way to prevent accidental github leaks? Don't use GitHub. Guaranteed to have 100% efficacy.
acdha 6y ago
Pre-commit.com has built in checks for AWS and SSH keys. I add this hook which checks for passwords, high-entropy strings, etc. and has tools for managing false positives: https://github.com/Yelp/detect-secrets https://github.com/Yelp/detect-secrets
montjoy 6y ago
Run your own git server behind a VPN. It still won’t prevent people from doing deliberately stupid things but will cut down on accidents. If you are super paranoid you air-gap everything.
BeefWellington 6y ago
Detect-Secrets, TruffleHog, or GitLeaks configured as a pre-commit hook in your local git repo to block you from committing them at all. Not perfect but gets you maybe 80-90% of the way.
0xbadcafebee 6y ago
This blog post by Duo Labs explains everything you need to know. It describes the problem in detail, breaks it down, and explains mitigations. Everyone should read this. https://duo.com/labs/research/how-to-monitor-github-for-secrets https://duo.com/labs/research/how-to-monitor-github-for-secr... Here is their comparison of pre-commit hooks and other tools. It's current and comprehensive. https://github.com/duo-labs/secret-bridge/blob/master/TOOLS.md https://github.com/duo-labs/secret-bridge/blob/master/TOOLS....
tialaramex 6y ago
Don't write keys into code, even "test" code. Discipline yourself (and your team) to use dedicated key storage even on test systems, (and of course that key storage isn't "Duh, a github repo" but an appropriate technology for your platform) If you instinctively want to write shiny_api_key = 'SOME.KEY.BOSS.EMAILED.ME' Force yourself to instead take the extra few minutes to do # my-cloud-vendor key-store create shiny_api_key SOME.KEY.BOSS.EMAILED.ME and then write shiny_api_key = key_store_access('shiny_api_key') This way, when six hours later you check that code in, to GitHub, and go home for the weekend, you won't get a call from the boss ruining your Sunday lie-in, "WTF Did you upload our $5000 shiny API key to GitHub? You are fired". This habit also allows for correct separation as an organisation grows, when you're big enough you want it to be possible for somebody to write code against test systems with test API keys, and that same code to work as written in production, even though production API keys are not accessible to the coders. Using key stores to mediate makes this much more likely to actually work.
- 0xbadcafebee 6y ago
  Don't know why you were downvoted, this is 100% what everyone should be doing. Don't even write shiny_api_key = "A-CLEARLY-FAKE-KEY". Don't write ANY secret AT ALL in your code, no matter what. Even for test frameworks. Just don't do it. Build in the secret retriving function from day one. Have it grab from an environment variable and it will be 100% seamless from development to production. Add more complexity/security if and when you need it. This can later be leveraged to allow for zero-downtime credential rotations for running services, which allows for seamless use of ephemeral credentials, which is what we should all be moving towards.
- yowlingcat 6y ago
  This is the wrong answer. At a company I worked at, we had a well known saying: "Favor mechanisms over good intentions." As the saying went, if good intentions worked, we "wouldn't need mechanisms." The right answer here (as stated in sibling comments) is to install a mechanism into the build. A push that accidentally commits credentials should break the build. How it goes about doing that is up to you, but the specific way I've done that in the past is a git hook [1] specifically the pre-commit hook. It looks like there's a reasonably well-updated package by Yelp to help do this kind of thing [2]. [1] https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks [2] https://github.com/Yelp/detect-secrets https://github.com/Yelp/detect-secrets
tetha 6y ago
Hm, we have multiple layers of protection of different kinds there, since defense should always be in depth. First off, I'm pushing the principle that secrets should only be issued, created, stored and distributed by the secret management system, vault in our case. If you never see a secret, and only instruct the automation how to handle it, it is much harder to leak the secret. Ideally, in production, no human should ever need to see or access the secret. Second, you should have good rotation procedures. Something will eventually leak, or someone will leave on bad terms and you need to be able to handle that. This can be simple - an HA infra service can just reload the core credentials from vault and be done, or somewhat harder, e.g. if customers need and use those passwords. And finally, you should have entropy scanners like trufflehog or github scanners as linked in other tools - both in you precommit hooks (so you cannot commit the secret) as well as in your CI & repository host, so someone who disabled the security scanner "because it was annoying" still gets caught. And I guess finally, whiskey for that friday evening someone still manages to leak something important. Some terminations at work have not been fun.
vangelis 6y ago
Using an external config source helps. It's harder to accidentally check in 'secrets.json' than an API key in code.

ardy42 6y ago

Sophistication isn't necessarily mean technical wizardry. IMHO, it's pretty sophisticated to thoroughly look for easily-exploitable goofs, and extract maximum value from them. The big advantage nation states have is probably mainly staffing and funding, and why use a zero-day when a leaked password will do?

porknubbins 6y ago
I can see how this could be called sophisticated, but worry about language inflation because we still need something to describe technical feats that are govt/military level like Stuxnet.
- ardy42 6y ago
  > I can see how this could be called sophisticated, but worry about language inflation because we still need something to describe technical feats that are govt/military level like Stuxnet. To start, I want to clear a potential confusion in my previous comment. By "technical wizardry" I meant "technical hacking wizardry," e.g. buffer overflows, ROP, etc. That said, I think Stuxnet and this hack may actually both be characterized by some of the same kind of sophistication I was referring to. IIRC, one of the biggest unique things about Stuxnet was the application of professional software engineering practices to malware. It wasn't a quick and dirty thing that one or two guys whipped up, it was a project, developed professionally over time by a team, to meet a set of requirements and allow code-reuse, etc. Not so much hacking wizardry but an application of government-scale money and personnel.
dx87 6y ago
Yeah, sophistication to me sounds more like the skill of the hacker. For example, logging in with leaked creds but cleaning up all relevant logs, and doing it during normal business hours for that account is a more sophisticated attack than using a zero day remote exploit and leaving a mess of evidence on the target systems.
deleted 6y ago
[deleted]
6y ago

wolco2 6y ago

Github is a single point of failure for the open source community

mistrial9 6y ago
> a single point of failure for the open source community some actors who benefit from scale are purposefully trying to make Github fit that role, to gather, search and mine the work of others without paying or giving back.. but due to the distributed nature of git itself, there are alternatives
creato 6y ago
What does this have to do with this incident? Do you think attackers having to scan 10 or more open source repo hosts for leaks is impractical? If so, maybe that's a worse situation? After all, if it's harder for attackers to scan, it must be harder for"white hats" looking for leaks to report? If anything, this might make companies increase the risk they associate with open sourcing their code. This wouldn't have happened if solar winds were not uploading anything at all to github...
- _jal 6y ago
  It isn't open sourcing that increases risk, it is the open repositories. It is a vital distinction to make. Code needs to be scanned for secrets even in closed repositories because of insider threats. But opening them up guarantees your mistakes will be caught by, ah, volunteer sysadmins. I understand you're pointing out that fuzzing the distinction is a way for someone with an anti-open-source agenda to advance their efforts, just trying to make the point explicit.

Nightshaxx 6y ago

Someone on HN called it before this broke. Governments are quick to blame "state-sponsored actors" to cover up for their own flawed security practices.

AndrewBissell 6y ago
The Washington Post immediately using the incidents to call for pushback against "Russia's secret war" doesn't inspire confidence in the official story either.

protocol_7 6y ago

Good time run to https://github.com/zricethezav/gitleaks https://github.com/zricethezav/gitleaks on your projects

danso 6y ago

I still think about how Instagram's entire kingdom (source code, SSL certs, AWS credentials) was left wide open because someone deployed an open source admin tool on a single server and never changed the default hardcoded secret key: https://news.ycombinator.com/item?id=10754194 https://news.ycombinator.com/item?id=10754194 https://web.archive.org/web/20151217205414/http://www.exfiltrated.com/research-Instagram-RCE.php https://web.archive.org/web/20151217205414/http://www.exfilt...

ProAm 6y ago
It's okay that person had to solve really tough programming interview questions on a whiteboard to get that gig.
- jesselawson 6y ago
  ^^ Most underrated comment here lol
  - WrtCdEvrydy 6y ago
    Sadly "The Whiteboard" doesn't test for having a pulse, just memorization :P
    - notReally01 6y ago
      Biologically speaking it’s all memorization. You memorized how to obfuscate comprehensions in a thousand ways. High 5 Discovery of net new abstraction is more interesting than re-proving you know what a mutex or cloud stack look like.
- deleted 6y ago
  [deleted]
- jennyyang 6y ago
  2-3 whiteboard interviews with 2 questions each. All questions need to be answered correctly.

tyho 6y ago

When reading press releases about cyberattacks, substitute "embarrassing" for "sophisticated" to improve the accuracy of the release.

mellosouls 6y ago
See also "nation state".
- phkahler 6y ago
  And substitute "teenager in his moms basement"?
  - mellosouls 6y ago
    That sounds about right.
  - erdos4d 6y ago
    Maybe. I'd bet a number of attacks attributed to nation states are actually just talented amateurs. There's a lot of smart people out there.
ChrisKnott 6y ago
The sophistication was in reference to the malware which made use of SolarWind's certificate, wasn't it? I don't see the need for this editorialised submission title.

bg24 6y ago

Sophistication is how the hackers hide their tracks and exfiltrate the stuff after they have come in. Coming in can be a simple stuff, or can be a zero day vulnerability.

SoylentOrange 6y ago

Are the authors suggesting the weak credentials were the only part of the attack? Or are they suggesting they were the first part in a rather sophisticated supply chain attack? The credentials by themselves don’t give execute privileges, or persistence, or in fact any access to non-SolarWinds systems. So both can be true.

wil421 6y ago
The author is just rehashing old events and making baseless speculations. Pure blog spam.

rahimiali 6y ago

it just looks like the upload credentials for an incoming directory were easy, but that doesn't imply a weakness. I can "upload" stuff to Google Drive. that doesn't mean Google is easy to hack.

rurban 6y ago
Not just the incoming ftp dir, the whole server was exposed, in and out. Plus the admin name and password

lock-free 6y ago

The article/twitter thread don't offer anything definitive. It could be this leaked password was how their supply chain got compromised, it might not be. Just anecdotally I'm looking at my own release infrastructure and it would take a bit more work to get compromised binaries to our customers than a leaked secret in a public git repo - and if it was reported that we leaked some key or password that gave the keys of the kingdom away, it would be trivial to change it and verify that we didn't push malware. This article can explain how malware made it into a download server, maybe. But does it explain the fact the malware was signed? Maybe, but that suggests a bigger fuck up than leaking a password.

hackerpain 6y ago
There was a case where we were able to gain access to Jira instance belonging to a bug bounty target (not even a Red Team engagement), and the Confluence wiki had every credential the company ever used. They used it as a secure credential storage. Majority of their internal panels didn't have a 2FA and we found internal VPN server credentials too (which didn't have source ip restrictions), so we could freely access tons of data. The bounty we got was around $1000 if I could remember, I think this would motivate a lot of hackers to not go the responsible disclosure way, as the incentives are ridiculously low. But I could access literally everything related to that organization and their clients from that single Github leak (of Jira creds).
- lock-free 6y ago
  Kind of an example of a "bigger fuck up" I was alluding to

lovecg 6y ago

This is just pure speculation at this point. The facts are that a leak existed and allegedly was fixed after the report. It’s quite a jump to conclude that this exact leak was used in the attack.

latchkey 6y ago

> For the readers who couldn't fully understand the tweet, Vinoth had apparently gotten access to a SolarWinds FTP server on 19th November, 2020 which is more than 1 year ago. 2019.

qw3rty01 6y ago

The email mentioned in the tweet is over a year old, with the response fixing the issue over a year old...the tweet in question is pointing out that their decisions relating to security are extremely questionable, not that this was the way they got in. This is also what he said in the Q&A session: > I think the attackers must have used the same approach as the FTP server was open & credentials were not strong enough. But it was a sophisticated attack as the binaries were signed.

julienchastang 6y ago

FTP? FTP does not have encryption (in contrast to HTTPS), correct?

notRobot 6y ago
There are multiple ways to secure FTP: FTPS and SFTP. FTPS uses SSL. SFTP is technically a different protocol, and uses SSH.
- julienchastang 6y ago
  From the screenshot, it appears they are using vanilla FTP?

lee990 6y ago

This article is literally pure-speculative garbage. You have one guy saying he found an FTP credential in github and on the other hand a nation-state attack and putting two and two together with no proof or reasoning other than they both happened at some point in time.

perennate 6y ago

The article title is "SolarWinds exposed FTP credentials in Public Github Repo" (in 2019) which is far less definitive than the misleading title submitted on HN.

jpdb 6y ago

This article is extremely dishonest and should be removed. It offers zero proof this is what lead to the compromise. The leaked credentials were reported over a year ago and patched according to FireEye. There are already comments here from users who presumably haven't read the article and are parroting this as fact.

tppiotrowski 6y ago
Agree. Also: "A malicious software class was included among many other legitimate classes and then signed with a legitimate certificate," the Windows maker said. The resulting binary included a backdoor and was then discreetly distributed into targeted organizations." - which means that somehow a signing certificate must have been stolen or reverse engineered. Not simple.
hackerpain 6y ago
The article has mentioned its purely based on speculation, it doesn't claim - it was the cause. There has to be a bigger chain of attacks that led to the breach.

DobbsMT 6y ago

Given the timing of the NSA announcement that Russian intelligence was actively exploiting the VMware-related CVE-2020-4006 just a couple of weeks before the Fireeye and .Gov announcements, I'm wondering if that was the source of the intrusion. The platform in question handles mobile device management, identity brokering, and mobile email management, among other things. It would certainly align with the reports about Treasury and other departments having email compromised. https://www.securityweek.com/russian-hackers-exploiting-recently-patched-vmware-flaw-nsa-warns https://www.securityweek.com/russian-hackers-exploiting-rece... “The exploitation via command injection led to installation of a web shell and follow-on malicious activity where credentials in the form of SAML authentication assertions were generated and sent to Microsoft Active Directory Federation Services (ADFS), which in turn granted the actors access to protected data,”

dfpojsdfpojonsp 6y ago

This reads like it was written by a bot. It's filled with sentence fragments and grammar errors.