4 ms·

What I want to know is the likelihood that customer devices were compromised. Frankly, I could care less about whether my data/password/whatever were lost in th

What I want to know is the likelihood that customer devices were compromised. Frankly, I could care less about whether my data/password/whatever were lost in the breach. Happens all the time, who cares. The problem with the Ubiquiti case is that this compromise has the potential to have poisoned official firmwares, with the silly source code/customer data bitcoin ransom as a smokescreen. Remember, there were VMs spun up in their cloud that they couldn't account for. That doesn't look like someone just tooling around to me, that looks like someone recreating a build process. Is anyone monitoring their Ubiquiti devices in their homelabs to watch for any suspicious traffic? Being part of a nation-state botnet is not a very comforting idea.

inetknght 6y ago
> Is anyone monitoring their Ubiquiti devices in their homelabs to watch for any suspicious traffic? With modern software good luck filtering suspicious noise from signal.
luma 6y ago
The problem is that UBNT now requires you to cloud-enable many of their modern devices as part of the setup process. You can later disable that after setup, but I think it's safe to assume most people don't know that (they certainly don't go out of their way to tell you that), and even fewer people will go through the effort. So, this hack also implies that a substantial number of devices out there now have their remote admin credentials leaked to an unknown, presumed malicious, third party.
- fossuser 6y ago
  If you have 2FA enabled are you protected from this? I suppose an automatic firmware update with bad code in it could still hit me?
  - cj 6y ago
    > If you have 2FA enabled are you protected from this? Very hard to say based on the limited information released. Part of what was released mentions "SSO cookie secrets" and "remote access". Best to assume 2FA isn't protecting you if the attackers were possibly able to forge cookies.
    - 1cvmask 6y ago
      The admin panel does not have 2fa support.
      - cj 6y ago
        When I managed a cloud key a couple years ago, login happened via SSO through ubnt, which has 2fa. (But yes, you can also login directly on the cloudkey locally which doesn't support 2fa) The concern is whether their remote login / SSO authentication via cloudkey was compromised.
      - tnorthcutt 6y ago
        ...this seems completely and totally insane
      - teilo 6y ago
        Yes it does, both on the local device, and on the Unifi cloud service. This is true as of recent firmware versions. But 2FA in this case is useless. The hackers owned the UBNT servers. If they harvested user logins, then one can assume that they also harvested the corresponding TOTP secrets. To make matters worse: When you setup a Gen2, Gen2+, UDM, or UDM-Pro, you are required to login to the UBNT cloud service to activate your device. Once you do this, your device now has an "Owner" account cannot be disabled. If you have enabled 2FA on your Unifi account, then you can now login to the device locally using your Unifi credentials + 2FA, with a nice false sense of security. So in other words, if your account creds and TOTP secret got leaked, and you did not change them, anyone who gains access to your local device's login page can still own you. This remains true even if you disable Remote Access. (And in fact, as of recent firmwares, the ONLY way to disable remote access is through the Unifi cloud panel. You can't even do it locally.) It will still call out to Unifi's cloud service and authenticate you. There's no way to turn off this SSO account. The best you can do is block all access to Unifi at the firewall, and download all firmware updates manually. And good luck with that if you are using a Unifi device as your gateway.
- WWLink 6y ago
  There is no way to disable cloud login to the unifi dream machine's web console. Not the network controller, but the web console itself. Like, the screen you get if you hit up your UDM's public IP, or go to the network controller (private IP) and open settings -> controller settings -> dream machine
- lsllc 6y ago
  I have 2 cloud keys that I only use for UniFi Protect -- mostly because they abandoned the UniFi Video that I could self host; my Network controllers are all containers that are mine (I do have 2FA set up for the UniFi Protect). I don't see how to remove the remote login from the Cloud Key, I tried creating a new Super-admin, but there's seemingly no way to remove the user. I guess I'll just have to factory reset them and start over with all new credentials.
count 6y ago
Not to discount the threat, but 'vms spun up in their cloud' is ...exactly what coinminining attackers do. It's so frequent, it's even called out as a 'thing to watch your bill to notice' in amazon security talks/etc. Gain access, launch N VM's, where N is as many as your credential and the cloud account limits in place allow, mine coins 'for free', and move on.
- ericbarrett 6y ago
  I agree, but APTs have gotten clever enough that I could also see them doing this as a distraction. Like burgling somebody's house to find blackmail material, but dressing it up as a common "smash & grab" theft so as not to raise their deeper suspicions.
- akersten 6y ago
  That's fair - and I hope that is the case that it's just mining. But I also hope Ubiquiti performs a full audit (ideally by an expert 3rd party) to ensure the integrity of their build systems and reassure customers that the worst hasn't happened. Because it's been radio silence on that part, and that they haven't even announced they're planning on it makes it seem like a blindspot.
carlisle_ 6y ago
>That doesn't look like someone just tooling around to me, that looks like someone recreating a build process. This is quite the leap in logic.
- akersten 6y ago
  It's speculation as I stated. But it's really not a "leap". The whistleblower's claim: > They were able to get cryptographic secrets for single sign-on cookies and remote access, full source code control contents, and signing keys exfiltration And what Ubiquiti has further stated: > we have well-developed evidence that the perpetrator is an individual with intricate knowledge of our cloud infrastructure. So someone with intricate knowledge of their infrastructure and access to their source code + signing keys, is doing what exactly with those VMs? I'm not trying to 'prove' that the firmware is compromised. We can't do that with the information we have. I'm just highlighting strong evidence that the possibility needs to be thoroughly investigated.
croutonwagon 6y ago
While my controller is on 6.0.43 and APs (uap-ac-lite and uap-ac-hd and uap-pro gen1) are on the corresponding firmware. I believe this controller version is December 2020 and predates the January issue disclosure date(though possibly not the incident). I have suricata taps on my net that are clean. Aps and controller are predictably trying to get to their telemetry call homes but nothing more. That said my Ubnt controller and aps have almost no internet access and haven’t for a long time. I only enable it when performing upgrades and disable it. I’ll pull the firewall logs and see specifically the IPs they are attempting to hit.

inetknght 6y ago

> Is anyone monitoring their Ubiquiti devices in their homelabs to watch for any suspicious traffic? With modern software good luck filtering suspicious noise from signal.

luma 6y ago

The problem is that UBNT now requires you to cloud-enable many of their modern devices as part of the setup process. You can later disable that after setup, but I think it's safe to assume most people don't know that (they certainly don't go out of their way to tell you that), and even fewer people will go through the effort. So, this hack also implies that a substantial number of devices out there now have their remote admin credentials leaked to an unknown, presumed malicious, third party.

fossuser 6y ago
If you have 2FA enabled are you protected from this? I suppose an automatic firmware update with bad code in it could still hit me?
- cj 6y ago
  > If you have 2FA enabled are you protected from this? Very hard to say based on the limited information released. Part of what was released mentions "SSO cookie secrets" and "remote access". Best to assume 2FA isn't protecting you if the attackers were possibly able to forge cookies.
  - 1cvmask 6y ago
    The admin panel does not have 2fa support.
    - cj 6y ago
      When I managed a cloud key a couple years ago, login happened via SSO through ubnt, which has 2fa. (But yes, you can also login directly on the cloudkey locally which doesn't support 2fa) The concern is whether their remote login / SSO authentication via cloudkey was compromised.
    - tnorthcutt 6y ago
      ...this seems completely and totally insane
    - teilo 6y ago
      Yes it does, both on the local device, and on the Unifi cloud service. This is true as of recent firmware versions. But 2FA in this case is useless. The hackers owned the UBNT servers. If they harvested user logins, then one can assume that they also harvested the corresponding TOTP secrets. To make matters worse: When you setup a Gen2, Gen2+, UDM, or UDM-Pro, you are required to login to the UBNT cloud service to activate your device. Once you do this, your device now has an "Owner" account cannot be disabled. If you have enabled 2FA on your Unifi account, then you can now login to the device locally using your Unifi credentials + 2FA, with a nice false sense of security. So in other words, if your account creds and TOTP secret got leaked, and you did not change them, anyone who gains access to your local device's login page can still own you. This remains true even if you disable Remote Access. (And in fact, as of recent firmwares, the ONLY way to disable remote access is through the Unifi cloud panel. You can't even do it locally.) It will still call out to Unifi's cloud service and authenticate you. There's no way to turn off this SSO account. The best you can do is block all access to Unifi at the firewall, and download all firmware updates manually. And good luck with that if you are using a Unifi device as your gateway.
WWLink 6y ago
There is no way to disable cloud login to the unifi dream machine's web console. Not the network controller, but the web console itself. Like, the screen you get if you hit up your UDM's public IP, or go to the network controller (private IP) and open settings -> controller settings -> dream machine
lsllc 6y ago
I have 2 cloud keys that I only use for UniFi Protect -- mostly because they abandoned the UniFi Video that I could self host; my Network controllers are all containers that are mine (I do have 2FA set up for the UniFi Protect). I don't see how to remove the remote login from the Cloud Key, I tried creating a new Super-admin, but there's seemingly no way to remove the user. I guess I'll just have to factory reset them and start over with all new credentials.

count 6y ago

Not to discount the threat, but 'vms spun up in their cloud' is ...exactly what coinminining attackers do. It's so frequent, it's even called out as a 'thing to watch your bill to notice' in amazon security talks/etc. Gain access, launch N VM's, where N is as many as your credential and the cloud account limits in place allow, mine coins 'for free', and move on.

ericbarrett 6y ago
I agree, but APTs have gotten clever enough that I could also see them doing this as a distraction. Like burgling somebody's house to find blackmail material, but dressing it up as a common "smash & grab" theft so as not to raise their deeper suspicions.
akersten 6y ago
That's fair - and I hope that is the case that it's just mining. But I also hope Ubiquiti performs a full audit (ideally by an expert 3rd party) to ensure the integrity of their build systems and reassure customers that the worst hasn't happened. Because it's been radio silence on that part, and that they haven't even announced they're planning on it makes it seem like a blindspot.

carlisle_ 6y ago

>That doesn't look like someone just tooling around to me, that looks like someone recreating a build process. This is quite the leap in logic.

akersten 6y ago
It's speculation as I stated. But it's really not a "leap". The whistleblower's claim: > They were able to get cryptographic secrets for single sign-on cookies and remote access, full source code control contents, and signing keys exfiltration And what Ubiquiti has further stated: > we have well-developed evidence that the perpetrator is an individual with intricate knowledge of our cloud infrastructure. So someone with intricate knowledge of their infrastructure and access to their source code + signing keys, is doing what exactly with those VMs? I'm not trying to 'prove' that the firmware is compromised. We can't do that with the information we have. I'm just highlighting strong evidence that the possibility needs to be thoroughly investigated.

croutonwagon 6y ago

While my controller is on 6.0.43 and APs (uap-ac-lite and uap-ac-hd and uap-pro gen1) are on the corresponding firmware. I believe this controller version is December 2020 and predates the January issue disclosure date(though possibly not the incident). I have suricata taps on my net that are clean. Aps and controller are predictably trying to get to their telemetry call homes but nothing more. That said my Ubnt controller and aps have almost no internet access and haven’t for a long time. I only enable it when performing upgrades and disable it. I’ll pull the firewall logs and see specifically the IPs they are attempting to hit.