10 ms·
Honestly, I think this should be the death knell of these "remote monitoring and management" tools that have extreme low-level access to networks and systems, b
by hn_throwaway_99 5y ago
Honestly, I think this should be the death knell of these "remote monitoring and management" tools that have extreme low-level access to networks and systems, but just like the SolarWinds attack, it feels like these are run by companies with extremely poor security culture.
I mean, I'd be willing to trust security to Microsoft or Apple (I mean, at some level, you've got to trust the OS). But giving the keys to the castle to some mid-tier company is just a recipe for disaster, and the bad guys know how extremely lucrative these targets are.
- gostsamo 5y ago> this should be the death knell of these "remote monitoring and management" tools Yeah, sure. We should have a person on each of hundreds of sites whose only job is to check manually every router, switch, and vending machine. Maybe in the best HN traditions you will train the necessary workforce in a weekend?
- elric 5y agoYour quote cuts off before the salient part, and you seem to be attacking an imaginary argument. > tools that have extreme low-level access to networks and systems The emphasis being on the low level access. The solution is not having hundreds of people checking things by hand (though I'm sure that could contribute to security). The solution is more privilege separation; so that when the "remote monitoring tool" is compromised, not every part of your infrastructure is also compromised by default.
- gostsamo 5y agoI'd agree partially with you. However, once a remote agent is compromised, it will be chained with some privilege escalation vulnerability and this same argument will be repeated with the twist that now every foreign executable with remote connection is an attack surface. Having hundreds of people in each location whose only task is to do a boring monitoring an very occasional management tasks is a waste of your resources and their intelligence. We do automation to escape doing stuff that we can but which are to mind numbing. The illusion that every one of those hundreds of people will do their job to the necessary level of quality and without lapses of diligence is optimistic to say the least. Doing automation badly is not a reason not to do automation at all.
- labawi 5y agoIMO, a big issue is conflating monitoring with management. Management is always going to have access, so maybe you should not enable remote management access of everything to a centralized system? Make it lean and secure, possibly segmented, dual-factor, use HSM etc. Monitoring - there is no good reason why it should have access to anything. Make it ingest only (use firewalls and reasonable protocols), and you've cut out most of the "monitoring and management" vulnerabilities.
- gostsamo 5y agoThere are trade offs there as well. Now you've decreased the attack surface, but still every foreign agent is a legalized rce. Observe that the case of Kaseya is not direct hacking of the agent, but a compromised update where firewall rules won't help. As I said, the next level of the argument is that this rce is dangerous and what it lacks is a privilege escalation. At the same time, you don't have a way to solve a problem that your monitor has alerted you for. Every solution proposed includes either a person at the location who does the job manually or a way to connect to the network from the outside which is vulnerable to similar attacks as before with added costs and possibilities to mismanage keys and passwords. Security vs convenience is a well-known dilemma that people very often love to solve in the most absolutist way.
- labawi 5y agoIf the software used a one-way protocol, then unless you updated the closed-source agents, which are the parts I have the biggest issue with, there wouldn't be RCE. As for remote management. I'm saying you should wisely choose what needs to be remotely managed, what doesn't, what are the foundations for your security and then balance it with reasonable methods to secure access. Which would probably not be "Kaseya VSA Remote Monitoring and Management" for all your systems and devices. Yes, make sure you don't need on-site personnel to restart your web server, but maybe also don't expose management of your switch that you never reconfigure to your monitoring SW, and maybe use separate HSMs¹ or at least HSMs instead of the enterprise management system for the most important parts. ¹ e.g. FIDO2 ed25519 for ssh
- greatquux 5y agoIt made my life as an MSP easier for sure and allowed us to support more clients and bigger businesses and get more done. But I fear you might be right, and it just isn’t worth the risk of a hole like this. Now I’m going to be up for days restoring servers, and data on any workstations that wasn’t backed up is gone. I think we’ll be filing a claim for this one.
- jiggawatts 5y agoJust a month or so after the attacks, one of our large government clients signed up to no less than three such vendors and deployed their products to almost all of their production servers. I discussed this with their security team leads, and they answered with a straight face that it's okay because they had to spend their budget before the end of the financial year.
- arp242 5y agoThis entire "we need to spend our budget"-attitude is something I will never understand. So what if you get less money next year? Firstly, it's not going in your pocket, and secondly clearly you can get by just fine with a lower budget. And everyone knows this is how it works too – so the Powers That Be keep setting the wrong incentives too. This is why I never worked for a large Enterprise company or government agency. I'd go crazy.
- yashap 5y agoAgreed. Companies that are great at selling to governments and massive enterprises tend to be great at security theatre and security certifications, but that’s not the same as being great at security. Their tech tends to be bloated spaghetti full of tech debt, with a huge surface area for attacks, and systems like that are nearly impossible to secure in a truly robust way. Embedding this kind of software deep in your internal networks/systems, with access to basically everything, is a recipe for disaster. I expect these sorts of supply chain attacks to get more and more common, they’re excellent back doors into basically every government agency and megacorp.
- colonelanguz 5y agoWould you mind briefly explaining the concept of "tech debt" to a layperson?
- oblio 5y agoWhen you cook, you make food, that's the primary result. You also produce waste, dirty dishes and general disorder. This secondary result is tech debt. You can cook a bunch of times ignoring these secondary results but over time cooking will be slower and of worse quality due to the mess and at some point it will be impossible (too dirty, no usable pots and pans, etc).
- DoreenMichele 5y agoIt's a quick fix that will take more time to fix later than it would to do it properly now but typically gets done to meet a deadline because time is of the essence currently for some reason. There is significant cognitive load in understanding the code and what it does and why it does it that way etc. Keeping all the important bits in one mind is challenging and a lot of the little fixes can lose sight of the big picture in a way that comes at a cost and, over time, this can really add up. Over time, different people may work on the code and have different reasons why they made different choices and at some point it may all stop playing well together. Then there comes a point where someone needs to try to reconcile all the different bits and understand what needs to happen and why and rebuilding the entire catalog of goals, features, etc. in one mind at one time so someone actually understands it all and gets it right is a substantial future cost that only grows as you keep delaying that step. (From one lay person to another -- I do write code, mostly html, and run some web projects, mostly blogs and Reddits, and spend too much time on HN. So technical debt isn't alien to my experience though I'm not really a programmer.)
- SV_BubbleTime 5y agoI specifically have experience with Kaseya. I kicked and screamed to get us off of it, the IT people insisted it was top notch. So when I became CFO I fired them (outside company), not just for this, but it didn’t help. It’s bad software. 24/7 full low level access is exactly what it is. We had an add on that stored admin credentials in a JSON… so looking back on that, it seems this should have happened sooner.
- daniellarusso 5y agoThe craziest part to me was their pushing of the vPro integrations.
- SV_BubbleTime 5y agoI dumped Intel servers for Epyc and Kaseya entirely. So, I’m looking pretty good with those decisions.
- briefcomment 5y agoDid you think it was an unintentional technical liability, or did you think it was intentional?
- SV_BubbleTime 5y agoUnintentional I think. If it was intentional I think things would have looked better on the surface.
- adolph 5y agoAdditional information about the Kaseya angle: https://doublepulsar.com/kaseya-supply-chain-attack-delivers-mass-ransomware-event-to-us-companies-76e4ec6ec64b https://doublepulsar.com/kaseya-supply-chain-attack-delivers...
- phendrenad2 5y agoI think that the problem is these companies are publicly-traded. Chasing YoY returns and never having a down quarter are antithetical to building a lasting security model.
- jonny_eh 5y agoMicrosoft, Apple, and Google seem to be doing ok.
- phendrenad2 5y agoThat's true, but when have FAANG unicorns ever had the same "laws of physics" that other companies have?
- daniellarusso 5y agoI used to work for an MSP and we had used Kaseya. There was an AV integration, and then Kaseya changed to Kaspersky. I don’t remember what the prior AV software was. I always thought it bizarre we were actively installing AV software from Russia on banking and medical office PCs.
- mjevans 5y agoThat has been a consideration in the AV software I recommend to friends, family, and professionally as an informal part of my threat assessment model. I viewed it as safer to buy products from anywhere other than someone that has ANY potential at all to go to war with the government of the country I live and work in. I really hope it never happens, but 'cold war' tensions might be waged with little cyber attacks and that software came to mind as a risk.
- SV_BubbleTime 5y agoOn the other hand… Kaspersky software isn’t shit. And I expect they catch a few things other US based companies might be incentivized or politely asked to look the other way on. I wouldn’t run K, but I know from experience it’s actually effective.
- viraptor 5y agoTwo more things to consider: - Can you articulate specific reasons to buy anything beyond the default windows defender? - If anyone went to an actual war with the US, would the source of your antivirus software get even close to top 5000 things you care about at that point...
- vel0city 5y agoAs for default Windows Defender, there isn't really good reporting tools related to it. There are reporting tools for Defender, but those are paid license add-ons. And yeah there's a decent chance if the US went to war with another country it might not impact the majority of US businesses very directly especially in the short term IRT their IT plans. McDonald's kept selling burgers when we invaded Iraq (multiple times). Ford was still producing vehicles during WWII. There have been lots of military engagements the US has been involved in where things in the mainland US weren't massively affected in day to day operations. Who knows what some potential future war with Russia would look like. Would it be a true head to head war with tanks rolling, fighter jets scrambling, cities bombed? Would it be more skirmishes testing how far the other would really go? Would it just be escalation of supply chain attacks and attacks on infrastructure to weaken the other? Of course this greatly varies based on the specifics on what that potential future war looks like, it would be naïve to think wars will always look like WWII, Korea, Vietnam, Iraq, etc from a US mainland perspective.
- cced 5y agoI know we're still pretty close to the Ubiquiti breach, but since then, they've added 2FA. Is your opinion of their products the same?
- aj3 5y agoUbiquity introduced new vulns while fixing that fiasco from last year: https://www.zerodayinitiative.com/blog/2021/5/24/cve-2021-22909-digging-into-a-ubiquiti-firmware-update-bug https://www.zerodayinitiative.com/blog/2021/5/24/cve-2021-22... On the other hand, all of the other networking HW sucks just as much. E.g. here are Netgear vulnerabilities published just this week: https://www.microsoft.com/security/blog/2021/06/30/microsoft-finds-new-netgear-firmware-vulnerabilities-that-could-lead-to-identity-theft-and-full-system-compromise/ https://www.microsoft.com/security/blog/2021/06/30/microsoft...
- beermonster 5y agoSome things, like updating firmware automatically, are ahead of their competitors. IMHO, the worrying things about Ubiquiti at the moment are: 1. Their handling of the security breach/downplaying/whistle blowing fiasco which came to light some months ago. Check our Troy Hunts podcast from around that time. 2. Requiring a cloud account to manage your local device. Everyone seems to do that these days. It's not impossible to remove the cloud account management but it is an extra post install PITA step to work-around. And has some consequences if you do. I'd like to see if they've learnt their lesson from at least the first point and become less opaque security-wise going forwards. Not sure their security is passing the smell test at the moment.
- atatatat 5y agoStatistically EVERYONE has extremely poor security culture. It's been wallpapered over as just cutting unnecessary expense for too long.
- machinehermitt 5y agoIt is almost proof we can't collectively think statistically. I get it at a pretty deep level individually but even knowing this I make enormous mistakes.
- perl4ever 5y agoWhat does "statistically EVERYONE..." mean to you? It sounds to me like if it means anything, it's denying that any probability depends on the exact dimensions of your ignorance. There is no statistic that applies to everyone, unless that person is a completely generic person with no known qualities.
- rjzzleep 5y agoA lot of these companies are actually huge enterprises with dozens if not hundred(s) of cybersecurity consultants and engineers. All of them are CISSPs and GICSPs(I do put my CISSP in the signature when working in those places too though). I go through security reviews all the time with them, they have so many security processes that you get dizzy and on paper everything looks fine. They create security zones with massive risk reviews, but for some reason those security zones then share subnets with the entire LAN. They also have a default configuration which makes everything access the standard intranet directory once its deemed secure. Enterprise security tools like Cyberark are deemed more secure than say yubikey HSMs, which may result in root ssh being enabled in a lot of settings. They have system configurations that are done with massive Excel sheets. Their cloud VPCs basically only have one risk profile and once its deemed secure it gets access to things in the intranet. They also vehemently refuse to do threat modelling when designing anything. These people can tell you so much about the theory of security by heart that it will make you dizzy but then won't actually understand the underlying problems. And the offenders are always the same, advised by Accenture, Infosys etc.
- tokamak-teapot 5y agoCan you explain more about the ‘root ash’ issue please?
- rjzzleep 5y agoBy using some of these tools they are under the false assumption that things that are otherwise considered security threats are somehow okay because for example the tool rotates passwords for you. It gives a false sense of security and allows you to do things that would otherwise be considered security threats. It's as if someone sells you a laser that shoots intruders and tells you, you can leave the front door open from now on, but that laser only works 1 in 3 times.
- tokamak-teapot 5y agoI see so if people have cyberark they might feel like it’s safe to enable root logins over ssh? That does sound like the sort of thing that would happen.
- ocdtrekkie 5y agoRMM is absolutely vital to securing systems. This is as ridiculous as suggesting we should just get rid of firewalls because there are vulnerabilities found in them. RMMs are how enterprise scale networks close off every other security hole on a network. That being said, RMM tools have plenty of examples that they need to beef up their security practices or get replaced.
- stonogo 5y agoNo, RMM is the magic beans someone wants you to trade your cows for. All OS and networking vendors have better tools, but people pay for RMMs because they make a lot of promises and charge less money. People who use them will invariably get burned.
- ocdtrekkie 5y agoThis is... laughably and demonstrably false. Neither Microsoft nor Apple tools even pass muster for large-scale IT management. Generally speaking, you'll pay more for a worse product direct from the OS developers. In most cases, your Microsoft-based solutions are less secure (RDP) than pretty much anything else. Apple is pretty much going to give you the minimum glue necessary to use a third party tool (see Apple Business Manager).
- stonogo 5y agoRDP is not a systems administration tool. Windows ADK is table stakes. It and its associated tools can handle any scenario from two-person startup to tens of thousands of employees. This is why Windows Commercial alone is a multi-billion dollar business, and these third-party RMM tools are pilot fish.
- rixed 5y agoActually, I think that the rapid blossoming of firewalls everywhere in the late 90s was the first big step toward security theatre that I noticed in my career, and illustrate how we have a cultural problem. Blocking access according to port number is no substitute for lack of control and visibility over what services are spawned where.
- ChuckMcM 5y agoOne could hope but I doubt it. CFO's gonna CFO and it's "cheaper" to outsource IT. I had one of these vendors really pushing me to "take a call" or "let them show me how they could cut costs". It was ALL about the costs. And I eventually called the CEO and said we would consider it if the company would take out a $100M bond that we could call on to repair any damage that occurred as a result of their managing our IT systems. He thought that was ridiculous of course and thought poorly of me. Since that time at least two of his customers have been the victims of breaches that IT either directly facilitated or indirectly made possible by providing additional attack surface that was required for their business to work. But not every person who has executive oversight of operations thinks like I do, and all of them are represented in the company's finances as a 'cost center' that is second only to Payroll in terms of how juicy a 'cost reduction' target it presents. So when the going gets tough, the company cuts back its IT budget.
- ipdd 5y agoThe CEO and CFO are right because this notion of reducing "attack surface" is not static. It changes from day to day and no technologist can guarantee what changes they add today makes any difference tomorrow. The promise is False. Therefore the principle of least action is justified.
- kaba0 5y ago“I will gotta leave open the windows and the door, and won’t even finish building the gates because I’m planning on extending the building and I will have to tore down a part either way”
- jart 5y agoI keep reading over and over again indignant comments about "cost centers" on Hacker News and I think it's not a good term to use because I looked up the definitions and the only logical consensus I could find is that everything which isn't shareholder profit is a cost center. It's just rhetoric.
- sgt101 5y ago
- cookiengineer 5y agoThe interesting part about last year's incidents of solarwinds, fireeye and fortinet is that there's a switch away from actually targeting the hosts after the first line of defense. Redteams / hackers now target the infastructure, because it's way easier and they're more outdated in regards of code, stability and used libraries. Most enterprise-grade VPN solutions still use OpenSSL from decades ago, and most of their fixes (even if they react to CVEs) are always too late. As SOCs need VPN access because they are usually not on-site, especially at larger corporations...the result is when you exploit the VPN gateway, you are the new administrator because you have a large time window until the SOC team arrives on-site. These couple hours are usually everything you need as a time window to raid the place, install and run ransomware, and clean up afterwards. From a cybersec perspective I cannot even begin to write how stupid it is to put literally all your company's value in the hands of a single security company - which is legally not responsible for anything by contract. Security through obscurity never worked, why should it do in this case? Last year showed that we desperately need an open source OpenVPN based graphical and scalable alternative that uses a standard TOTP based token generation mechanism and not some proprietary crap for authentication.
- fragileone 5y agoWireGuard is a simple and secure new protocol that most VPN companies are moving to. It doesn't do the key rotation or TOTP authentication part however.
- cookiengineer 5y agoMost of the attacks the last years were also targeting the enterprise auth apps that were heavily outdated (e.g. vasco token apps like "Auth ES" or "Enterprise Auth" etc). Lots of the breaches could've been prevented by just using a standardized (maintained) TOTP token generator that doesn't have an RCE backdoor with its included analytics scripts. Using a token generator with embedded analytics was just wrong in the first place, but...yeah. Personally I'd love to see better Wireguard support and adoption outside the Linux world.
- staticassertion 5y ago> But giving the keys to the castle to some mid-tier company is just a recipe for disaster It sucks, because I know my company is quite small but we take security extremely seriously (we have 9 people, 4 are security engineers, and the other 5 have varying degrees of experience in security). I think people might worry that, because of our size, we won't be as secure as a larger company. But the irony is that larger companies are often far less secure than us, because we've done shit right from day 1. There's just not a lot of ways to prove it. Compliance is meaningless. You could get a pentest report, but it really comes down to who's doing the pentest, and so if your pentest becomes a public doc the incentive is to have them go easy on you - not to mention that lots of reports contain "findings" that are nonsense but a casual reader might misunderstand. We plan to give talks and blog about how companies at our stage can do things that would make companies 100x our size jealous, because that's kinda the only thing we can do to really explain that it's possible. I think it's totally criminal that companies ask for RCE on all of your devices and then push out some closed source C++ app that's probably parsing all sorts of random shit, reading poorly authorized commands from some C2, etc.
- shandor 5y ago> We plan to give talks and blog about how companies at our stage can do things that would make companies 100x our size jealous Sounds interesting and rather extraordinary, would be great to read more on your thoughts on that. Do you refer to the Grapl blog?
- staticassertion 5y agoYeah, I'd say watch the blog, I have some draft posts written up.
- PenguinCoder 5y agoMuch easier to do it Ina small company, very hard to get it right in a company "100x your size".
- staticassertion 5y ago
- Vaderv 5y agoThank you M$FT
- _the_inflator 5y agoYes, and there is a reason why big IT providers like Accenture are preferred enterprise vendors. They have the financial power to mitigate such risks. There are usually vendor risk checks which include potential damage costs.
- aiisjustanif 5y ago> I'd be willing to trust security to Microsoft Well then I guess we will have a lot threats from hackers for days to come.
- ThePowerOfDirge 5y agohttps://arstechnica.com/gadgets/2021/06/microsoft-digitally-signs-malicious-rootkit-driver/ https://arstechnica.com/gadgets/2021/06/microsoft-digitally-...
- INTPenis 5y agoWhat if Apple and Microsoft out sorces to some mid-tier company? That's what is happening in my business, the big telco/consultancy business. We're always out sourcing things and there have been several scandals related to these small to mid-tier companies that get only a small part of the contract.
- whoisthemachine 5y agoUnfortunately, these tools are so damn useful that people almost feel compelled to buy and use them. Eventually, as these attacks becomes more and more commonplace, I think companies will start looking for two things: 1) How secure is the software? Where are the audits? 2) If your software compromises my business, how much of the losses I incur as a result will you cover?
- NoImmatureAdHom 5y agoYou do not have to "trust" the OS at some level. Use Linux or BSD, demand open hardware. You only feel like you "have to trust" shitty closed-source OSes because the orgs behind those OSes have been able to abuse market-dominant positions to stifle competition. Security by obscurity is laughable nonsense. We should all be demanding transparency in hardware and software from our vendors. I'd pay handsomely for it.
- tgragnato 5y agoI think that in this context the meaning of the term trust is different. Any code executing in privileged mode can bypass security, and is therefore inherently part of a system's trusted computing base (TCB). (Linux is a monolithic kernel running in ring 0) Most companies are not Linux contributors, they are trusting the kernel developers to write bug free, secure code. Minimizing the TCB and opting for an auditable open source TCB are really useful concepts in security. But the cause of these breaches is much more trivial than what you are worrying about: these companies are basically installing whichever piece of software can decrease their costs without thinking about what they’re doing.
- NoImmatureAdHom 5y agoI agree with you. >Any code executing in privileged mode can bypass security, and is therefore inherently part of a system's trusted computing base (TCB). (Linux is a monolithic kernel running in ring 0) It's way nicer to be able to look at the code running in Ring 0 =)
- scrozart 5y agoAgreed. I can't imagine outsourcing monitoring/metrics/etc, despite the mild hassle of maintaining our server of one of the popular options. It requires attention every now and then, like once a year or two, but can be integrated easily with LDAP and our SSO provider.
- syshum 5y agoYou know it is ironic to say you trust MS when they are in the middle of a major security incident themselves i.e "PrinterNightmare" for which they still have not patched
- wglb 5y agoAnd I am curious to know how many such vendors there are.