13 ms·
Even with the best intentions, can a volunteer-driven project like OpenSSH truly guarantee the same level of security as a commercial solution with dedicated re
by davidfiala 2y ago
Even with the best intentions, can a volunteer-driven project like OpenSSH truly guarantee the same level of security as a commercial solution with dedicated resources and a financial stake in preventing backdoors?
- chgs 2y agoBetter. Imagine a closed source company with cost pressures employing a random developer who can commit code, perhaps without any peer review, but certainly limited peer review from harried employees. Now imagine why a nation state would want to get staff working in such a company. Now if companies like Microsoft or Amazon or Google want to pay people to work on these open source projects that’s a different thing, and a great thing for them to do given how much they rely on the code.
- wannacboatmovie 2y agoYour argument is a model that does no vetting of contributors whatsoever, which resulted in the catastrophe that is the topic of discussion, is better than a hypothetical company which is full of compromised developers that have free reign to commit to the source tree with no oversight? That sounds extremely contrived.
- adolph 2y agoIf you are positing that government infiltration of companies is hypothetical and not a real threat, here is an example of compromised corporate staff: https://en.wikipedia.org/wiki/Saudi_infiltration_of_Twitter https://en.wikipedia.org/wiki/Saudi_infiltration_of_Twitter
- chgs 2y agoThis wasn’t a contributor to OpenSSH, it was a deep level supply chain attack - something that closed source commercial companies are not immune to. Given how much closed source companies love BSD/apache/etc licenses where they can simply use these low level libraries and charge for stuff on the top I’m not sure how they would be immune from such an attack. The risk from this was highlighted in xkcd back in 2020 https://xkcd.com/2347/ https://xkcd.com/2347/
- wannacboatmovie 2y agoMoving the goalposts and splitting hairs. The fact remains the open source model allowed an imaginary person, operating on behalf of a threat actor, to obtain privileged commit access to a widely used open source project without any vetting whatsoever. Let me repeat that. They were given control of the repo without even verifying this person exists. To do this at a commercial company you actually have to show up and interview which is an order of magnitude more difficult than creating an anonymous Gmail account and be given the keys to the kingdom.
- anthk 2y agoYou are the one who moved the goalpost here. Vanilla OpenSSH doesn't link against xz, period. Not even the portable versions as LibreSSL does for OpenSSL. If distros randomly patch OpenSSH because of SystemD, it's their problem.
- davidfiala 2y agoThere's a ton of great truth here. It's hard to bite the bullet and believe that insiders already exist (everywhere), but I can share that from my experience working in big tech: - There 100% will be bad actors. Many of them. - But not always nationstate. Instead, they do it for (dumb) personal reasons, too. Also, don't forget lulzsec as a great example of just doing it for fun. So we cannot presume to know anything about the 'why'. The bad guys I caught did it for the most asinine reasons... But the good news is that we have options: - Strategic: Develop processes and systems that account for the perpetual existence of unknown bad actors and allow for successful business operation even when humans are compromised. - Reactive: Structural logging that makes sense in the context of the action. Alerts and detection systems too. - Reduction: Reduce access to only what is needed, when it is needed. - Proactive (not always necessary): Multi party approvals (a la code review and production flag changes or ACL changes, too) - Social: Build a culture of security practices and awareness of bad actors. Don't make people feel guilty or accusatory, just empower them to make good design and process decisions. It's a team sport. Bonus: By guarding against evil actors, you've also got some freebie good coverage for when an innocent employee gets compromised too! --- Companies like Google and Amazon do the techniques above. And they don't generally rely on antiquated technology that cannot and will not change to meet the modern standards. I know because I was the person that built and Google's first time-based access system and rational-based access systems. And multi party approval systems for access. (Fun fact: The organizational challenge is harder than the technical). And, those strategies work. And they increase SRE resilience too! --- But even with the best UX, the best security tooling, the best everything, etc there's no guarantees that it matters if we just reject anything except the old system we're used to. It's like a motorcycle helmet: Only works if you use it.
- jasonjayr 2y agoHow can a commercial solution prevent backdoors? A sensitive product like this would have to defend against well funded, patient, well resourced threats, including but not limited to infiltrating an organzation in order to plant code that only a few people may even be able to notice.
- 2OEH8eoCRo0 2y agoWell for one they need to show up in person. They can't be some anon anime character who hides their identity for totally legitimate reasons.
- dijit 2y agoIt's extremely easy for a three-letter agency or similar to plant a new employee. Corporate espionage may not be talked about very much, but it is still very fashionable. Even without state sponsored attackers.
- deleted 2y ago[deleted]
- toast0 2y agoAs an employee, I've typically needed to show up in person, but I've worked with contractors who never showed up in person. I've even been such a contractor at times. Lots of commercial products use contractors and licensed code in the final product. At least with most open source projects, a lot of the contribution process is in the open, so you could watch it you wanted to. As DonHopkins writes elsewhere, few people do, but it's possible. Not a lot of commercial projects offer that level of transparency into changes.
- kstrauser 2y agoI worked at my current job for 3 months before I met a coworker in person. That might slightly help at a legacy butts-in-seats factory, but doesn't do a lot for remote jobs. I could be proxying in from Romania for all they'd know.
- yjftsjthsd-h 2y agoThankfully, we aren't limited to asking leading questions and then hand waving at it; we have a rather lot of empirical evidence. OpenSSH is 24 years old; has it ever been successfully backdoored?
- davidfiala 2y agoWe don't know. We won't know the negative case, but we may someday in some circumstance find out the positive (bugged) case. But we do know some sane things: - The stakes couldn't be higher. - Good: Don't allow inbound SSH connections, even through a fancy $100k firewall. - Best: Don't let people login with SSH (treat SSH like we treat the serial port: a debugging option of last resort)
- tptacek 2y agoWho's running OpenSSH through "fancy $100k firewalls"?
- davidfiala 2y agoIt's off topic, but in my consulting and networking, security/firewall appliances are an easy first line approach I see companies buy in to. The security sales pitch sounds good and makes you feel good. Cannot name names.
- tptacek 2y agoI mean, everybody has a perimeter, even the ZT believers, but I think the notion of large networks protected by like a high-end NetScreen or Palo Alto firewall is 10-15 years out of date. We have, like, Tailscale, and netfilter.
- aflukasz 2y agoRe "good"/"best": you are thinking about air gapping, then? Pull based systems are susceptible as any other software.
- dijit 2y agoReality has shown that the least secure systems tend to be: A) The ones with financial stakes in the game. combined with: B) Completely closed systems. Contrarily, the most secure seem to be the ones volunteer led, with no financial stakes. It doesn't matter what you think is true, this is clearly what is consistently happening.
- davidfiala 2y ago> tldr; your statement overlooks the reality of businesses with high ethical and financial obligations, like Google, Amazon, and Azure. - These companies underpin much of the internet's infrastructure. - Their security practices are far more advanced than typical businesses, with SSH being a heavily restricted last resort. That's not to imply that everyone else shouldn't strive to do meet that (modern) bar too. - Dedicated teams focus on minimizing access through time-based, role-based, and purpose-based controls. - They actively develop new security methodologies, often closed-source, but with public evidence of their impact (e.g., https://cloud.google.com/docs/security/production-services-protection https://cloud.google.com/docs/security/production-services-p... ). - They rarely experience conventional hacks due to reduced blast radius from attacks and insider threats. - Leading security experts in both major tech companies and niche organizations are driving new strategies and ways to think about security... their focus includes access reduction, resilience, and reliability, regardless of whether the solutions are closed or commercial for them. The ideas spread. (looking at you, Snapchat, for some odd reason) - This is key: This evolution may not be obvious unless you actively engage with those at the forefront. I think it's what makes people think like the comment above. We cannot see everything. - It's crucial to recognize that security is a dynamic field... with both open-source and closed-source solutions contributing. So, the notion that volunteer-led projects are inherently more secure overlooks the significant investments in security made by major corporations that host the internet, and their relative success in doing so. Their advacements are coming to the rest of the world (eventually).
- dijit 2y agoVolunteer led still seems to be more secure, even with a lot of corporate investment. Rather than corporate lead endeavors which are very hit and miss, mostly miss, especially when the product itself claims security as a core principle. It might not make sense to you, but the evidence points to this.
- toast0 2y agoOf course not. OpenSSH comes with no warranty, read the license. Historically, it's been pretty good though. If you would consider a commercial alternative, consider how much you would need to pay to get an actionable warranty, and consider if you could find someone to do a warrantied audit of OpenSSH (or especially of OpenSSH in your environment) for that amount. It might be a better use of your money.
- dns_snek 2y agoDid you forget to disclose that you're a founder of a YC-backed commercial solution that wants to compete with SSH?
- cirrus3 2y agoDavid Fiala CEO & Founder at Teclada Inc | Former Google Security Leader | Supercomputing PhD https://www.linkedin.com/company/teclada?trk=public_profile_topcard-current-company https://www.linkedin.com/company/teclada?trk=public_profile_...
- davidfiala 2y ago@dns_snek, it's right there in my real name username, comment history, and profile. :) My entire youth and professional life I've seen nothing but footguns with actual practical use of SSH. The HN community loves to hate, but the reality is that almost no one uses SSH safely. It's near impossible. Especially when it comes to configuration, keys, passwords, and network factors. I observed the common SSH failure patterns, and I made the most obvious footguns less than lethal. Looking a step further, I made remote terminal access a pleasure to use safely even for absolute novices. So to your point about being in YC: In doing so, I thought it would be beneficial to join a community that supports one another (YC) so that an option (Teclada) can scale to make a real impact in the world WRT the warts and footguns of SSH.
- kstrauser 2y agoAsking genuinely, what common footguns do you see? What are the usual failure patterns?
- ChoHag 2y ago[dead]
- LinuxBender 2y agoNot the person you are asking but the two footguns I see all the time in corporations are the use of multiplexing in SSHD and the mismanagement of public key trusts in authorized_keys. I suspect this may also be an issue in the DoD given none of the federal hardening guidelines address this so I hope they are reading. Especially right now Multiplexing: This feature improves speed when using SSH to proxy applications but it also gives an authentication-free unlogged channel to phishers, allowing zero friction trivial bypass of MFA. There are ways to log this with auditd but very few companies even touch auditd much less customize it to catch phishers. With multiplexing and phishing, corporate firewalls effectively become non existent and phishing is highly effective even in places one would think it would not be. Phishing capabilities include but are not limited to any org with access to public email or public chat, Slack, Instagram, Signal, Whatsapp, Discord, etc... Loose Lips Sink Ships Public Keys: People instinctively worry about private keys but the biggest risk in OpenSSH in my opinion is the lack of auditing of what public keys are created by whom or what and trusted on what accounts. I can for example add a key to your account if I have temporary root privs and the much later log in to a system as you do bad things and now you are the first person investigators look at. Combine this with passwordless-sudo and it's game over, not that one really needs root to pilfer secrets from a company or let competitors destroy it. In most companies this is a hot steamy mess that people choose to ignore and do mental gymnastics to avoid it all together. How to do these things properly is a much bigger topic, too big for HN comments.
- deleted 2y ago[deleted]
- whydoyoucare 2y agoWe must first precisely define "level of security" that is expected from OpenSSH and a commerical version. Only then the discussion about who can guarantee what would make sense.
- dessimus 2y agoHas any open source project taken down the majority of single OS install base as quickly as CrowdStrike? Seems like they would have the "dedicated resources and a financial stake" to prevent such as situation.
- tptacek 2y agoOpenSSH, as load-bearing infrastructure for much of the Internet, is heavily sponsored by tech companies. It empirically has one of the best security records in all of general-purpose off-the-shelf software. If for some benighted reason I found myself competing directly with OpenSSH, the very last thing I would pick as a USP would be security.
- SoftTalker 2y agoUSP is Unique Selling Proposition?
- tptacek 2y agoYep. I would not attempt to differentiate against OpenSSH based on security track records. It's one of the most trusted pieces of software in the industry.
- GJim 2y agoWhy the hell is a genuine question being downvoted? Downvoters, what are you trying to achieve?
- udev4096 2y agoHN audience is full of hypocrisy and ignorance
- gmuslera 2y agoSometimes commercial companies have "incentives" to put backdoors, like i.e. secret orders by intelligence agencies. Snowden papers and all related information from that time set a baseline on what you may consider safe.
- mmsc 2y agoJuniper had a backdoor in their firmware in 2015 which gave ssh access, inserted by hackers: https://blog.cryptographyengineering.com/2015/12/22/on-juniper-backdoor/ https://blog.cryptographyengineering.com/2015/12/22/on-junip... There were some updates: https://www.zdnet.com/article/congress-asks-juniper-for-the-results-of-its-2015-nsa-backdoor-investigation/ https://www.zdnet.com/article/congress-asks-juniper-for-the-... and according to https://www.reuters.com/article/world/spy-agency-ducks-questions-about-back-doors-in-tech-products-idUSKBN27D1DO/ https://www.reuters.com/article/world/spy-agency-ducks-quest... China was behind it (but without evidence).