21 ms·
> The attacker gained access to the server — which had been active for about a month — by exploiting an insecure remote management system left by the datacenter
by LennyWhiteJr 7y ago
> The attacker gained access to the server — which had been active for about a month — by exploiting an insecure remote management system left by the datacenter provider, which NordVPN said it was unaware that such a system existed.
This screams for clarification and I'd love for someone more knowledgeable in the area to elaborate on it. Is this common practice for data-center providers? Do I now not only have to worry about my own infrastructure security but also worry that my IaaS provider hasn't installed some backdoor to my servers?
- CiPHPerCoder 7y agoSounds like an iDRAC exploit (assuming Dell servers). But, yes, remote management is pretty common in datacenters. The fact that NordVPN wasn't aware of them just shows incompetence.
- lucb1e 7y agoDepends on what kind. In case of idrac, yes; but it's weird that it was insecure by default in the first place. Usually credentials are configured and provided to the customer. Makes me think there might have been some other interface. Clarification is definitely needed.
- CiPHPerCoder 7y agoIt could have been something like https://www.zdnet.com/article/vulnerabilities-found-in-the-remote-management-interface-of-supermicro-servers https://www.zdnet.com/article/vulnerabilities-found-in-the-r... There were many IPMI/iDRAC/etc. exploits published in the past few years. Throw a dart at a list of them, and you'll probably find one that was unpatched in most systems as of March 2018.
- LennyWhiteJr 7y agoHow the hell do you pwn a server with iDRAC?
- berbec 7y agoIn certain configurations, iDRAC gives you an rdp connection. If idrac is left at default, windows admin login not being changed isn't too much of a stretch.
- zeta0134 7y agoIf you can reboot it without anyone noticing? Really, really easily: iDRAC gives you access to the local console, like a remote KVM. Reboot into single user mode, change a password, done.
- notyourday 7y agoOh, IPMI and friends are a total mess. Some implementations allow one to take control of a running server remotely especially if they use a shared ethernet for management ( popular in supermicros ). I once had our security geek demonstrate it by taking over the running server, rebooting it using network emulated USB stick, adding a file into /etc and rebooting the server again. In secure environments one pulls IPMI module from the server or only uses the modules that have their own dedicated NICs that have to be wired to their own management network.
- tonyarkles 7y agoThe first time I booted a server using a virtual CD-ROM (iso on my laptop shows up as a hardware CD-ROM on the server) over IPMI I was simultaneously relieved (because I could fix the machine remotely) and absolutely totally horrified.
- kevin_b_er 7y agoiDRAC is a full onboard whitehat rootkit manufactured and supported by Dell. It runs independently of any OS and has control over the system. It is intended to be a substitute for physical access.
- luch 7y agoiDrac have a default password : https://danblee.com/dell-idrac-default-username-and-password/ https://danblee.com/dell-idrac-default-username-and-password... There also quite a number of sysadmins that connect idrac's to the "regular" network, instead of the sysadmin VLAN ...
- _wldu 7y agoUser root, password calvin. That's the default. And, if I had a dime for every time I've seen one of these in a data center, I'd be a rich man. I have literally begged sys admins to change the default password, but they say, "Why... we're behind a firewall using RFC 1918 addresses. No one can get to these." The rest, as they say, is history.
- WrtCdEvrydy 7y ago> we're behind a firewall This is the dumbest thing I've ever seen... unless your firewall is between your host versus every other host and there's no multi-tenancy, this will suck.
- allset_ 7y agoIn well maintained networks the management interface (IDRAC, etc.) for each server is placed on a separate VLAN which the servers cannot access. This isn't to say that cheap providers actually do this, or that the VLAN can't be accessed by a compromised technician's workstation/laptop.
- fulafel 7y agoSo it's a fail-open design, given the rarity of well maintained networks, and the lability and inobservability of said state. Never trust the network.
- jeltz 7y agoYes, this kind of firewall is always supposed to be between the management hosts and everything else. Only the sysadmins at the data center a very limited set of applications is supposed to be able to access it. The very real risk is misconfiguration.
- pm7 7y agoHPE iLO also had critical vulnerability: CVE-2017-12542.
- ryanlol 7y agoI’d guess that the DC got owned, no need for iDRAC exploits when lazy VPN company staff never changed the pws.
- atmosx 7y agoPointing fingers without having the details at hand is not competent either.
- RL_Quine 7y agoThey mean IPMI. All servers have IPMI and there’s remote root exploits against many versions of them.
- laumars 7y ago> All servers have IPMI Most (not all) servers have out-of-band management; of which IPMI is just one of many such solutions. It's also worth noting that the hack could have been against in-band management if the Nord used an OS image provided by the DC hosts. However OOB feels more likely given their description (as vague as it was).
- stu2010 7y agoIf they're going with a bottom-dollar host, it's possible that the out-of-band server management tools were exposed. It's less likely to be a software backdoor, and more likely to be Supermicro IPMI or other baseboard management controller.
- oarsinsync 7y agoYes. Dedicated servers generally have IPMI / ILO / IDRAC / whatever. It's the only way to scale out management of hosts and provisioning.
- oarsinsync 7y agoTo the person / people down voting, can you share your ideas for how you can go about providing at-scale management of large estates of physical machines? I've never seen anything else that's both as practical and as affordable as IPMI / IDRAC / ILO systems that ship with servers, that doesn't introduce weird new failure conditions that can impact significantly more than a single host.
- ProAm 7y agoOf course you have to worry about all of that! When you don't self host you have to assume whoever you are renting from hirers the lowest paid employees they can get to manage infrastructure for you. That's how they get profitable. You are not outsourcing expertise.
- hipsterstal1n 7y agoI know that public cloud providers like Rackspace and Azure insert their own accounts and services into cloud servers and VMs mostly under the guise of being able to support said servers and monitor them and their health. True data centers where you own the hardware shouldn't... they give you an ethernet cord and everything is on you.
- notyourday 7y agoVery few people go to "true data centers". Those are very expensive because you are buying power, space, cooling and cross connects. Racking machines, replacing hard drives, building a PXE-boot infrastructure, building a remote access infrastructure that bypasses the customer facing network is expensive and time consuming.
- LIV2 7y agoIt doesn't make much sense to me, even with iDRAC/some other console access you don't really have access to OS unless you reboot & go to single user mode etc at which point they should be noticing their servers rebooting etc. would love more info
- russiandriver 7y agoIt opens an exploit chain, in a normal circumstance you are correct. In a malicious circumstance, it is always feasible irrespective of the likelihood.
- slrz 7y agoJust set up your code as a boot-once config and wait for the owner to reboot their machine. Make your code end by booting the installed OS (or even by just rebooting again, most people will just curse about the damn slow server boot process).
- MertsA 7y agoYou can't do that as you don't have any access until it's being rebooted. It's basically like you're standing in front of the machine so there's not really much you can do when you're just looking at a login prompt, you have to be able to stop grub from just booting with the default options and instead boot up using init=/bin/bash or maybe if the server supports iPXE you can just chain load some payload off the internet.
- slrz 7y agoYou can manipulate boot settings using BMC commands. No need to mess with Grub or the running system. Instead, tell the system to boot up from an emulated USB drive (image can be attached from some remote server, often including your web browser). Now wait for the machine to get rebooted (or do it yourself using the BMC, e.g. 'racadm serveraction powercycle' for Dell/iDRAC machines). Even SecureBoot won't help as you can just turn it off using the BMC. See here for a bunch of examples for Dell machines using the BMC's HTTP API: https://github.com/dell/iDRAC-Redfish-Scripting/tree/master/Redfish%20Python https://github.com/dell/iDRAC-Redfish-Scripting/tree/master/...
- jugilefo 7y agoNot sure if it is still the case but a while ago it was standard on OVH servers for them to put some public keys in their root-like's user authorized_keys. I think they used that to perform tasks requested from the web management system.
- StreamBright 7y ago>> Is this common practice for data-center providers? Absolutely. We had similar situation with one of the DC vendors.
- yread 7y agoSo what can you do about it?
- StreamBright 7y agoUse AWS/GCP/Azure
- dajohnson89 7y agodoesn't that shift the vulnerability/risk to the configuration of the firewall, security policy, and machine configuration?
- StreamBright 7y agoThose cloud computing vendors know better than give open access to the word to your infra by default.
- woadwarrior01 7y agoFull Disk Encryption is an option, but that’ll entail needing the use the IPMI console to enter the password at every reboot, essentially turning it into a manual operation.
- zeta0134 7y agoI work for a web hosting company in the US and at least in our case, it's quite common for remote management to be enabled on pretty much all of our dedicated hardware. However, because of the inherent dangers in opening this up to the public internet, unless explicitly requested by the customer (or Managed Colocation), the NIC used for Dell iDRAC or HP iLO is on an isolated network unique to the physical datacenter. Remote access for our techs is managed through a secured bridge that requires all sorts of security hoops on our company intranet, and remote access for general internet traffic is not available due to the firewall restrictions. While it's plausible for remote access to be gained this way, it is extremely unlikely and would require several exploits at different points along the path. I cannot speak for the industry as a whole, but remote management systems like this are bound to be common; any large enough physical datacenter is going to need a more efficient way to access a misbehaving system than sending a tech physically running to the box to plug in a keyboard and mouse. It should be extremely uncommon to have these management interfaces open to the public though, and I'll bet that's what NordVPN is surprised by. Generally these systems should be private and isolated due to the power that an attacker can wield through them.
- buildbuildbuild 7y agoIPMI does not have to be open to the internet to be open to a wide audience. Many of these out of band management interfaces are hosted on an internal network, but not isolated by customer. Cheap datacenters are favored by VPN providers for their unlimited bandwidth and lax abuse policies. Many of them allow access to IPMI only over a VPN, but do not isolate each customer’s IPMI to a customer VLAN. I personally know at least three large budget datacenters which allow all customers access to each others’ “private” IPMI IP addresses.
- djsumdog 7y agoYes, network KVMs are expected of any co-location center. You want to be able to access the console and the power switches of any real physical server without having to send someone out to the center, and is a common feature of most high end data centers. Even a lot of VM/cloud systems have some kind of virtual management console (Linode has their LISH system that lets you SSH in to console and Vultr/Digital Ocean have similar web based consoles .. AWS surprisingly doesn't. You can get console output but can't send VMs any console input). Not only should have NordVPN been aware of this hardware KVM, they should have secured it and had version checks on its firmware as an essential part of their security. I could see this oversight with other companies, but not with one whose primary business claims to be security.
- notyourday 7y ago> Yes, network KVMs are expected of any co-location center. You want to be able to access the console and the power switches of any real physical server without having to send someone out to the center, and is a common feature of most high end data centers. Power on/off should be done via APIs that issue commands to a PDU, like Atlantic.net started doing in the early 200s. And there's nearly zero reason to access "console" - configure your server to always but off PXE and fall through to disk if that intercept is not needed.
- jessaustin 7y ago.. AWS surprisingly doesn't. Why is this surprising? AWS seem to know what they're doing in general, and this is obviously the right policy in this particular area.
- comex 7y agoHow is it “obviously” the right policy? If implemented, console input would presumably be part of the AWS API, guarded with IAM permissions like everything else. If you have full IAM permissions, you can already take over any instance by temporarily attaching its disk to a different instance and modifying the data from there. (That requires rebooting, but so would takeover via console input.) Indeed, I’ve had to do exactly that on multiple occasions to fix broken config files on my personal instance; it would have been much more convenient to have console input.
- badrabbit 7y agoLeaving default creds on IPMI/iLO is not uncommon,some providers don't allow internet access,you have to login on the web console and use a java applet to access it iirc. I can't imagine a well reputed provider exposing ipmi to the internet but the nature of their business means they have to diversify server and network providers.
- halbritt 7y agoTypically data centers have compliance requirements like SSAE 16 specifies controls around physical access. Most any major retail data center would have that certification and others. One presumes that because of NordVPN's business, they're colocating a server or two in many very many "POPs", presumably not all of them have tight controls on physical access. Its likely that there are none available in many areas where they seek to maintain a point of presence.
- deleted 7y ago[deleted]
- illuminati1911 7y agoIt seems NordVPN fucked up themselves are are now trying to avoid responsibility: https://www.theregister.co.uk/2019/10/21/nordvpn_security_issue/ https://www.theregister.co.uk/2019/10/21/nordvpn_security_is... ""All servers we provide have the iLO or iDRAC remote access tool, and as a matter of fact this remote access tool has security problems from time to time, as almost all software in the world. We patched this tool as new firmware was released from HP or Dell. "We have many clients, and some large VPN service providers among them, who take care of their security very strongly. They pay more attention to this than NordVPN, and ask us to put iLO or iDRAC remote-access tool inside private networks or shut down access to this tool until they need it. We bring [iLO or iDRAC] ports up when we get requests from clients, and shut them down when they are done using this tools. NordVPN seems it did not pay more attention to security by themselves, and somehow try to put this on our shoulders.""
- wazoox 7y agoOn Supermicro hardware and maybe others, IPMI has a very dangerous default setting: if you're not connecting the dedicated IPMI port to a network (typically some closed network dedicated to management), it will use the first ethernet NIC on the motherboard (sharing it with the host), possibly making it accessible through the internet (with default, insecure credentials adding insult to injury) or at least neighbouring machines.