8 ms·
SolarWinds leaked FTP credentials through a public GitHub repo since 2018
- sschueller 6y agoIt doesn't take much of a "sophisticated hack" if they post their passwords publicly.
- deleted 6y ago[deleted]
- hackerpain 6y agoThe news and press defined the "attack sophistication". An attack of this scale definitely needs some amount of sophistication but on a difficulty scale of 1-10 to infiltrate their network, I would rate it a humble 0.5/10.
- ceejayoz 6y agoDon't blame the press on this; the "sophisticated hack" wording comes directly from SolarWinds. https://www.solarwinds.com/securityadvisory https://www.solarwinds.com/securityadvisory > We have been advised that this incident was likely the result of a highly sophisticated, targeted, and manual supply chain attack by an outside nation state...
- hackerpain 6y ago> We have been advised Their advisor was smart enough to call it so.
- ceejayoz 6y agoAny "sophisticated hack" is likely to have some mundane details. This might've given an avenue to start exploring; it doesn't seem likely it's what let them sign binaries by itself, and it doesn't change the cleverness of the steganographic techniques described in the FireEye analysis.
- martinald 6y agoThis isn't really a mundane detail though. The fact they have non secure (not even SFTP or FTP over SSL) FTP access open to their main downloads page with a ridiculously insecure password is pretty startling. I can't even understand why they had FTP running like that.
- ceejayoz 6y agoHaving worked with large enterprise vendors, it's really not startling to me. Depressing, perhaps. I'm saying, though, that a hack can be sophisticated even if the initial compromise was something pretty basic like this. As an example, https://about.fb.com/news/2020/11/bug-bounty-program-10th-anniversary/ https://about.fb.com/news/2020/11/bug-bounty-program-10th-an... > Earlier this year we received a report from Selamet Hariyanto who identified a low impact issue in our Content Delivery Network (CDN), a global network of servers that deliver content to people accessing our platform around the world, where a subset of our CDN URLs could have been accessible after they were set to expire. After fixing this bug, our internal researchers found a rare scenario where a very sophisticated attacker could have escalated to remote code execution. As always, we rewarded the researcher based on the maximum possible impact of their report, rather than on the lower-severity issue initially reported to us. It is now our highest bounty – $80,000.
- hackerpain 6y agoI have found a LARGE social media having a similar kind of vulnerability. Downloaded some data as Proof of concept. There's no way to reach their CEO and no security contact. How should I report it in your opinion?
- ceejayoz 6y agoDepends on the site, the data involved, your jurisdiction, their jurisdiction, etc. In the EU, you might be able to report it to a regulator. In Russia, perhaps not.
- mackenzie-gg 6y ago
- acdha 6y agoIf you read the reports, rather than second or third hand coverage, note how the attacker concealed their activities after they got in. That’s what sophisticated refers to - breaches are common but avoiding detection for so long on so many networks is not.
- neffy 6y agoThat last is actually rather hard to determine. Detected attacks tend to divide into two groups - the scripted continuous probing that's just background noise these days (but if you have access to the logs on any internet connecting device in front of a firewall you will typically see it being probed at least every 10 minutes), and targeted attacks - where the first rule is to be very careful, lay low, and perform careful data exfiltration. Reading the reports on the kinds of the latter that were discovered, and in particular how they were discovered, suggests there is probably much more of that going on than is generally realised.
- rst 6y agoIt's not just the initial compromise that's getting "sophisticated hack" headlines in this case -- it's the extraordinary care that the attackers took to evade detection afterwards. Among other things: delaying communication to home base for a couple of weeks after the backdoor was initially installed; use of steganography to make that look like ordinary network traffic afterwards; use of entirely new exploits, not in any common database, to avoid detection triggered by a match. The FireEye report describes all of this in more detail: https://www.fireeye.com/blog/threat-research/2020/12/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with-sunburst-backdoor.html https://www.fireeye.com/blog/threat-research/2020/12/evasive...
- jjk166 6y agoWhile the specific implementations in this case were unique and clever, these tactics aren't particularly uncommon. If you have direct access to SolarWinds' signing credentials and download server, the rest is just a matter of how much time and effort the attacker is willing to put into it.
- KukiAirani 6y agoI assumed that all software in the supply chain for defense would go through an audit that exposes things like this. Did that never happen?
- LinuxBender 6y agoThey might ask if password complexity/rotation is enforced. They probably don't ask if any accounts (service accounts, etc..) are excluded from rotation. People are also taught to not volunteer anything to an auditor. ... and this does not even get into the subject of ssh keys and ssh CA's, which most auditors don't even ask about at all. I personally put less value in audits and more value in Red Teams being given full immunity to penetrate every nook and cranny. I would like to see more companies incentivize and reward in-depth penetration testing in all environments, including production. For the corporate leaders reading this, there is risk, but the reward is uncovering many future landmines your operations, code deployment teams and internet user base would have stepped on.
- RegnisGnaw 6y agoThis is true. At my $PREVIOUS work place, we answered all questions from auditors and RFPs using the most favorable interpretation towards us of the question. For example, one of our audits for our product asked if we implemented admin idle session timeout with timeout of less than 15 minutes. There was 3 admin panels: one with 10 minute idle session timeout, one with 1 hour idle session timeout, and one with no idle session timeout. The manager said after I explained the 3 admin panels to him something like "since one does have idle session timeout of less than 15 minutes, we answer yes to the audit question".
- kryogen1c 6y ago> we answered all questions from auditors and RFPs using the most favorable interpretation towards us of the question. I am currently living this life. The problem is that the system is setup as a race to the bottom with opposed incentives. If I answer with the strictest interpretation with my paranoid blue-team attitude, we appear worse than our competitors and are immediately in a worse business position - regardless of our relative or absolute security posture. This is why the Department of Defense is moving away from self-attestation in 800-171 to outside assessment in CMMC. Why not standardize on ISO and SOC2? I dont know very much about it, but I suspect those big-boy standards arent suitable for small-business America/sub-subcontractors
- EvanAnderson 6y agoThis is egregious, for sure, but it doesn't explain how a DLL signed with their certificate ended up in the wild. Have SolarWinds' handling practices for their code signing certificate come to light? It's sounding more and more like we're going to find out it was a "PFX file w/ the password 'password' saved on a network share" kind of situation.
- jjk166 6y ago> This raises many questions. Were the certificates used to sign the binaries obtained from that public GitHub repository or, from any other information leaked publicly. Exposed certificate could have allowed hackers to sign their malicious SolarWinds Orion binaries and pass them off as legitimate software developed by SolarWinds, subsequently uploading them to the Downloads server with the previously found leaked FTP credentials.
- roland-s 6y agoThis is wild speculation but just to highlight how FTP could be leveraged to get a signed DLL. Imagine SolarWinds has a "sign everything in this directory" script. Attackers gain access to that trusted directory either directly via FTP or by pivoting off FTP access. They just plop their DLL in there and let SolarWinds auto-sign it. No cert required.
- EvanAnderson 6y agoScary as heck but possible. I implemented an HSM-based signing service for a firmware attestation system a few years ago. Authorization to sign and an audit trail of signature requests were a big deal. Something like the "plop a file in a directory" would make me weep.
- ohiovr 6y agoStolen certificate is my guess.
- markus_zhang 6y agoI love the word "since" in the title.
- earthboundkid 6y agoWait, FTP as in actual goddamn FTP, or actually SFTP but we call it FTP? I could make fun, except in 2018, I moved a company off of FTP and onto S3. To be fair to the company, no developer had worked there since 2016, so they were just running on autopilot. Still, anyone even vaguely concerned with security should have stopped using FTP since sometime in the early 00s.
- deleted 6y ago[deleted]
- worstenbrood 6y agoYou can actually use ftp with implicit/explicit ssl, makes it as save as any other plain text protocol (smtp/imap/pop/...). They call it FTPS.
- lub 6y agoTLS doesn't necessarily mean everything is encrypted with FTP, because data and control channel have to be encrypted independently. See for example https://security.stackexchange.com/a/115565 https://security.stackexchange.com/a/115565 or https://en.wikipedia.org/wiki/FTPS#Secure_data_channel https://en.wikipedia.org/wiki/FTPS#Secure_data_channel for more detailed explanations. Most of the time both are encrypted when configuring TLS, but it's not as easy as with IMAP or SMTP where you basically disallow all commands except STARTTLS.
- worstenbrood 6y agoIf you use implicit tls/ssl, not using starttls, the data channel should be using tls/ssl too.
- ThePadawan 6y ago> Still, anyone even vaguely concerned with security should have stopped using FTP since sometime in the early 00s. Had the same discussion in 2018. But if the weak link in the chain is "our (paying) customer is so tech illiterate that they need a printed out tutorial to use an FTP client, and switching to SFTP would anger all clients", what are you going to do.
- netsharc 6y agoConsidering how often people commit their credentials, it seems Git or GitHub need a --i-need-a-nanny option that would check if you're about to publish things that you would not rather publish. But then it would have to be an installation option because the people who goof up like this would not know/remember to add that CLI parameter; but I would bet a lot of people would turn on said nanny option that would warn you if you're about to commit lines containing the word "password" or "secret" or "key" (obviously it would have to be more clever than this simple text compare, otherwise it would warn on lines like "function checkPassword(String userSuppliedPassword) {"...
- ceejayoz 6y agoGithub already does this to some extent. https://docs.github.com/en/free-pro-team@latest/github/administering-a-repository/about-secret-scanning https://docs.github.com/en/free-pro-team@latest/github/admin...
- eli 6y agoThere are lots of tools that do that. Obviously it’s always going to have false positives and false negatives. A dev environment that makes it easy to deal with secrets the right way is probably better
- fjfaase 6y agoThere are also ways of using Git without a repository in the cloud. As a security oriented company, it seems a good policy to avoid using cloud services as much as possible.
- SamBam 6y agoGitHub already warns you if you accidentally check in an API key. It's like a rite of passage for new devs at my company setting up a new project. Then we have to walk them through changing the keys.
- blntechie 6y agoSomeone in my company committed an AWS credential to GitHub public repo and AWS actually emailed within few hours saying that the access key has been deactivated or something alike. AWS seem to be scanning the public repos pretty actively for such instances.
- willis936 6y agoDoes github have a log of IP addresses that cloned the repo? Not that an IP address has much information.
- nwsm 6y agoAlmost certainly. But people who are scraping GitHub for leaked credentials are almost certainly doing so from IPs that would be hard to link back to them. Through VPNs, tor, compromised devices, etc. Not to mention that you'd be tracking down people who may have not (yet) done anything illegal.
- ChrisMarshallNY 6y agoThis type of thing can happen easily with Git. We just submit a directory, and every damn file in that directory gets submitted. What I do, is have a file that aggregates my sensitive stuff (like server secrets and whatnot), and call that file something like "DoNotCheckThisIntoSourceControl.swift". I then add a git ignore line, on that name. I'll also sometimes store it outside of the repo root (I use Xcode, so I can drag files in from anywhere).
- hackerpain 6y agoNot if you use TruffleHog and GitGuardian.
- ChrisMarshallNY 6y agoThose are "close the barn door after the horses have run away" solutions. The best solution is to plan ahead.
- mackenzie-gg 6y agoYou can put detection in the CI/CD pipeline to prevent from getting into the repository. And in any case. Knowing the horses have run away as soon as possible is pretty essential in damage prevention. What is interesting for me here is that this leak, like the Brazilian covid leak, happened because of an employees GitHub repository. Which companies have no authority over. GitGuardian at least scans the GitHub accounts of employees though.
- ChrisMarshallNY 6y agoTrue, but I have always taken the position that security starts before we write our first line of code, and should be something that we keep front and center, every line we write. When I write a line of code, I have an automatic "red team" approach. I think "I'm a blackhat hacker. How do I leverage this line?" Not a 100% guarantee, but it sure does help the result to be more secure. In my experience, 99% of security is good old-fashioned horse sense.
- abarringer 6y agoAlso this --> Security researcher Vinoth Kumar told Reuters that, last year, he alerted the company that anyone could access SolarWinds’ update server by using the password “solarwinds123” https://www.reuters.com/article/global-cyber-solarwinds/hackers-at-center-of-sprawling-spy-campaign-turned-solarwinds-dominance-against-it-idUSKBN28P2N8 https://www.reuters.com/article/global-cyber-solarwinds/hack...
- bulatb 6y agoThis was posted yesterday and flagged into oblivion. Unless new information has come out, there wasn't and still isn't any reason to believe this leak had anything to do with 2020's megabreach. Edit: To clarify, this version follows up the flagged post with a bit more information and a lot more speculation. It does a lot of work to "not" make claims while setting them up and basically making them. https://news.ycombinator.com/item?id=25419140 https://news.ycombinator.com/item?id=25419140
- hackerpain 6y agoThere's some new information released which mentions the github repo name, and since when the vulnerability has existed.
- bulatb 6y agoYeah, this is a less-but-still-misleading follow-up. The password might turn out to be related or might not, but I don't think the added details point either way. This security company's marketing blog is mixing up the facts with speculation. Let's especially be careful not to use it in the "nation state vs. kid in a basement" debate.
- hackerpain 6y agoThe researcher added there may be some certificates exposed in that repo which may have been used to sign the binaries. It's still a relevant update. Especially the information that the repo was archived by Web Archive back in 2018. It's not easy to know the "who" but the how can be speculated and investigated.
- bulatb 6y agoI think I snuck an edit in while you were writing. Sorry! I see where the researcher says it's not impossible he missed a certificate, but not a reason to believe he actually did. The article is unfortunately full of leading questions and reaching speculation.
- vesche 6y agoI discussed this in a video I made on the SolarWinds compromise if anyone is interested - https://youtu.be/ONd0ERCUy0k?t=760 https://youtu.be/ONd0ERCUy0k?t=760 Original Tweet came from @vinodsparrow - https://twitter.com/vinodsparrow/status/1338431183588188160 https://twitter.com/vinodsparrow/status/1338431183588188160 Keep in mind the binary files that contained the backdoor were digitally signed by SolarWinds after being tampered with. So this FTP credential leak might be part of the supply chain compromise, but is not the whole enchilada.
- mbag 6y agoHere is one interesting project that lets you see in almost real time leaked secrets (or suscpected secrets there might be fasle positives) across Github, Gists, Gitlab, and Bitbucket: https://www.shhgit.com/ https://www.shhgit.com/ You can also run your own instance: https://github.com/eth0izzle/shhgit/ https://github.com/eth0izzle/shhgit/
- hackerpain 6y agoIt's useful but yeah a ton of false positives :) Most common false positives appear to be the test values.
- giantg2 6y agoGlad I'm not the ASC for that product
- sombrero_guitar 6y agoHighly sophisticated attack that only nation state could have accomplished or leaked FTP credentials that any script kiddie could have used?