3 ms·

What I’m curious about is, if I run my own controller on my own hardware, do I need to be concerned about this? I could understand supply chain concerns... I’ve

by teeray 6y ago

What I’m curious about is, if I run my own controller on my own hardware, do I need to be concerned about this? I could understand supply chain concerns... I’ve held off updating anything while this plays out. But all these “breach! breach!” stories fail to spell out who is affected and what they need to do.

gerdesj 6y ago
You probably don't need to be concerned(ish). I run a controller for 32 "sites" across the UK with 1 to 13 APs per site and a few switches. I keep it behind HAProxy but with fairly minimal changes (from memory.) I have stuck with controller 5.13.32 rather than moving to 6.x just yet. It's an LTS version and I'm still waiting for the whinging to stop on the forums. I also watch the AP firmware and that has had some interesting times over the last few months. I've confirmed dodgy AP versions on my sites and backrevved and held accordingly. I treat the whole thing the same way I do any other system. I come out in spots when people mention clouds and IT in the same sentence, so I have not knowingly enabled any cloudy integrations from my controller to UBNT. Specifically, I have not enabled "Remote Access".
ev1 6y ago
Force pushed updates overnight turned local controllers into requiring ui.com single sign on, iirc.
Nextgrid 6y ago
If the compromise is widespread enough then the attackers might have gained control of the update infrastructure allowing them to push out malicious firmware to your devices.
- js2 6y ago
  These blanket statements don’t apply to everyone. It depends which Ubiquiti hardware you own and how you’ve configured it. For example, I run the UniFi controller on my FreeNAS server. There are no forced updates to it. It doesn’t update unless I update it. The firmware on my APs doesn’t update unless I update them from my controller.
  - ncphil 6y ago
    Agreed. My only gear is an EdgeRouter-4. Unlike the Mikrotik it replaced you have go up, find the latest fw file, download and install (that Mikrotik router wasn't designed to handle 1 Gbps and at the time the next step up cost more than the ER).
    - lucb1e 6y ago
      So unless it hits news channels major enough that you hear about it or there is a bug that you isolate to be due to outdated firmware, you probably won't ever patch security issues in your edge (outside-facing) router?
      - ncphil 6y ago
        I've got a recurring calendar item that prompts me to go up and check for updates every week, and I do what it says because after over 25 years in sysadmin I _know_ that the first time I ignore it I'm going to get stung. With Mikrotik I originally had auto update turned on, but later had a script that emailed me when an update was available (so I could schedule a manual update a couple of days later after checking the forums for anything apocalyptic).
  - izacus 6y ago
    Unless you're manually verifying the content of your AP firmware updates (which is a bit hard since they're closedsource), I don't understand what you're trying to say. The firmware could be compromised at the source so your FreeNAS doesn't help at all when you download and apply a compromised firmware update. Unless you're not updating your APs and keeping them vulnerable in that way :)
    - js2 6y ago
      I was addressing "attackers might have gained control of the update infrastructure allowing them to push out malicious firmware to your devices." In my case, no, they cannot push anything to my devices. Obviously, I could pull down compromised firmware. But that's always a risk with software that I don't personally verify, which is like 99.9% of software. As a side note: obviously this security incident doesn't give me a whole lot of confidence in how they run their systems, but at no point has it been alleged that Ubquiti's firmware updates have been tampered with.
izacus 6y ago
If you read the original post, the they noticed a breach when someone put an "unknown" VM on their server infrastructure. The attackers also got signing keys for firmware. So even if you run a local controller, I see two very serious vectors: 1. The "Ubiquiti account signin" functionality - you probably had it off, but I'd like a confirmation that it doesn't keep a backdoor open anyway. 2. Having a malicious firmware update put on the servers. If it took months for someone to find the vulnerability, who knows how long the servers could push a compromised controller/firmware builds for the hardware.
- elric 6y ago
  The self-hosted controller UI uses your browser to fetch crap from UI.com. Mine just launched a request to net-fe-static-assets.network-controller.svc.ui.com/videos/empty, which I'm sure is perfectly reasonable...I can think of a thousand legit reasons why my network controller would need to load, erhm, an empty video. sigh I wish I could trust them.

by teeray 6y ago

gerdesj 6y ago

You probably don't need to be concerned(ish). I run a controller for 32 "sites" across the UK with 1 to 13 APs per site and a few switches. I keep it behind HAProxy but with fairly minimal changes (from memory.) I have stuck with controller 5.13.32 rather than moving to 6.x just yet. It's an LTS version and I'm still waiting for the whinging to stop on the forums. I also watch the AP firmware and that has had some interesting times over the last few months. I've confirmed dodgy AP versions on my sites and backrevved and held accordingly. I treat the whole thing the same way I do any other system. I come out in spots when people mention clouds and IT in the same sentence, so I have not knowingly enabled any cloudy integrations from my controller to UBNT. Specifically, I have not enabled "Remote Access".

ev1 6y ago

Force pushed updates overnight turned local controllers into requiring ui.com single sign on, iirc.

Nextgrid 6y ago

If the compromise is widespread enough then the attackers might have gained control of the update infrastructure allowing them to push out malicious firmware to your devices.

js2 6y ago
These blanket statements don’t apply to everyone. It depends which Ubiquiti hardware you own and how you’ve configured it. For example, I run the UniFi controller on my FreeNAS server. There are no forced updates to it. It doesn’t update unless I update it. The firmware on my APs doesn’t update unless I update them from my controller.
- ncphil 6y ago
  Agreed. My only gear is an EdgeRouter-4. Unlike the Mikrotik it replaced you have go up, find the latest fw file, download and install (that Mikrotik router wasn't designed to handle 1 Gbps and at the time the next step up cost more than the ER).
  - lucb1e 6y ago
    So unless it hits news channels major enough that you hear about it or there is a bug that you isolate to be due to outdated firmware, you probably won't ever patch security issues in your edge (outside-facing) router?
    - ncphil 6y ago
      I've got a recurring calendar item that prompts me to go up and check for updates every week, and I do what it says because after over 25 years in sysadmin I _know_ that the first time I ignore it I'm going to get stung. With Mikrotik I originally had auto update turned on, but later had a script that emailed me when an update was available (so I could schedule a manual update a couple of days later after checking the forums for anything apocalyptic).
- izacus 6y ago
  Unless you're manually verifying the content of your AP firmware updates (which is a bit hard since they're closedsource), I don't understand what you're trying to say. The firmware could be compromised at the source so your FreeNAS doesn't help at all when you download and apply a compromised firmware update. Unless you're not updating your APs and keeping them vulnerable in that way :)
  - js2 6y ago
    I was addressing "attackers might have gained control of the update infrastructure allowing them to push out malicious firmware to your devices." In my case, no, they cannot push anything to my devices. Obviously, I could pull down compromised firmware. But that's always a risk with software that I don't personally verify, which is like 99.9% of software. As a side note: obviously this security incident doesn't give me a whole lot of confidence in how they run their systems, but at no point has it been alleged that Ubquiti's firmware updates have been tampered with.

izacus 6y ago

If you read the original post, the they noticed a breach when someone put an "unknown" VM on their server infrastructure. The attackers also got signing keys for firmware. So even if you run a local controller, I see two very serious vectors: 1. The "Ubiquiti account signin" functionality - you probably had it off, but I'd like a confirmation that it doesn't keep a backdoor open anyway. 2. Having a malicious firmware update put on the servers. If it took months for someone to find the vulnerability, who knows how long the servers could push a compromised controller/firmware builds for the hardware.

elric 6y ago
The self-hosted controller UI uses your browser to fetch crap from UI.com. Mine just launched a request to net-fe-static-assets.network-controller.svc.ui.com/videos/empty, which I'm sure is perfectly reasonable...I can think of a thousand legit reasons why my network controller would need to load, erhm, an empty video. sigh I wish I could trust them.